Partisci’s documentation

Where is your software installed? Is version X still in use anywhere?

Partisci answers these questions by collecting updates from your programs and providing a REST API to access the data.

Partisci can answer these questions:

  • What hosts is application A installed on?
  • Which versions of application A are active?
  • Is version V of application A still active anywhere?
  • Which hosts are runinng version V of application A?
  • When did application A last update? (from host H?)
  • What applications are installed on host H?

However, Partisci only knows about applications which send updates. Each application needs a small change to send these updates.

Details

Some limitations:

  • Applications must be modified to send updates.
  • If applications are not running, no information is updated.
  • Partisci only logs the most recent data fro each host/app/instance combination. In the future, Partisci may keep more historical data.
  • It is intended to be used within an organization to track custom software. It has no features to support publicly released software, or anything your organization can’t modify.

Partisci accepts update updates via UDP and HTTP. Sending UDP updates is recommended. When UDP isn’t easy or possible, use HTTP (for example: network restrictions or browser based apps). Partisci requres each application to be modified to send these updates.

Updates contain:

  • application name
  • version
  • hostname
  • instance number

Partisci also logs:

  • client address (as seen by Partisci)
  • time the update is recieved

Where does the name come from?

  • It is a truncated portmanteau from Partially Omniscient. Within its limited domain, Partisci strives to be omniscient.

Commands

partiscid

partiscid is the server. For a quick test:

$ partiscid -v
2012/02/14 08:10:02 Starting.
2012/02/14 08:10:02 listening on: 0.0.0.0:7777
...

This will start the server, listening on port 7777, using memory to store version updates.

partiscid also supports the following flags:

$ partiscid -help
Usage of partiscid:
  -danger=false: enable dangerous commands for testing
  -listenip="": listen only on this IP (defaults to all)
  -port=7777: listening port (both UDP and HTTP server)
  -sqlite="": use SQLite for persistence and store db at this path
  -trim=0: keep Versons until this many seconds have passed, then discard
  -v=false: log more details

The -trim option will remove references to any instances which have not sent a version update within the number of seconds specified by -trim.

partisci

partisci is a command line utility to communicate with partiscid.

The main command is update, which sends a single custom update message:

$ partisci update "Demo App A" 1.0 host1.example.com 0
command: UPDATE

partisci also supports the following flags:

$ partisci -help
Usage of partisci:
  partisci update APP VERSION [[HOST] INSTANCE]
Options:
  -port=7777: partiscid port
  -server="localhost": partiscid address (IP or DNS name)

Clients

Partisci ships with clients for several languages.

python

The pypartisci module implements an simple client which supports updates via either UDP or HTTP. It does not support the full REST API.

The most applications should send updates via UDP:

import pypartisci

SERVER = "localhost"
PORT = 7777
APP = "Python UDP Example"
VER = "1.0"

if __name__ == "__main__":
    pypartisci.send_udp(SERVER, PORT, APP, VER)

If an application needs to be sure a version update is recieved, HTTP is also supported. If the server is unreachable for any reason, this function will raise an exception:

import pypartisci

SERVER = "localhost"
PORT = 7777
APP = "Python UDP Example"
VER = "1.0"

if __name__ == "__main__":
    pypartisci.send_http(SERVER, PORT, APP, VER)

Both functions also accept host and instance parameters:

send_update(SERVER, PORT, APP, VER, host, instance)

The PORT and instance variables should be integers, all others are strings.

go

The golang client supports sending updates via UDP:

package main

import (
	"log"
	"github.com/briandorsey/partisci/clients/go/client"
	"github.com/briandorsey/partisci/version"
)

func main() {
	v := version.Version{App: "app", Ver: "1.2.3",
		Host: "example.com", Instance: 0}
	err := client.SendUDP("localhost", 7777, v)
	if err != nil {
		log.Fatal(err)
	}
}

API

Partisci provides access to the data it collects via a REST API.

The REST API also inclues an update resource for situtaions where applications cannot update via UDP.

Partisci can answer the following questions:

  • Which hosts is application A installed on?
  • Which versions of application A are active?
  • Is version V of application A still active anywhere?
  • Which hosts are runinng version V of application A?
  • When did application A last update? (from host H?)
  • What applications are installed on host H?

All of the following urls are rooted at /api/v1/. Ex: app/ is at /api/v1/app/.

verb path description
GET app/ distinct active applications
GET host/ distinct active hosts
GET version/ every A & H with their most recent version
GET version/?app_id=A version for every H running A
GET version/?host=H version for every A on host H
GET version/?app_id=A&host=H version for app A on H
GET version/?app_id=A&ver=V version for app A, version V
POST update/ accepts a version update body
GET debug/vars process statistics (memory use, etc)
— only when running in -danger mode
POST _danger/clear/ clear the entire version database
— items below not implemented yet
GET / overview

Version JSON

A version update has the following JSON structure:

{
  "app" : "Application Name",
  "ver" : "1.2.3dev",
  "host" : "hostname",
  "instance" : 0,
}

app and ver are required. instance defaults to 0 and host will be inferred from the IP connection if not specified.

app, ver & host are limited to 50 unicode characters & instance is an integer 0-65535 (uint16).

When returned from Partisci, the following additional fields will be added:

"app_id" : "application_name"
"host_ip" : "10.0.0.1"
"last_update" : 1327940599

Where host_ip is the IP address of the sending machine as seen by Partisci and last_update is a unix epoch time stamp, rounded to the nearest second. app_id is a simplified form of app for use in referring to the application in the REST API.

All version/ urls return a full version JSON object.

Update clients

Clients should send update packets via UDP. Update packets are raw UTF8 encoded bytes containing the version JSON.

For clients which cannot use UDP, they can post the version JSON to the update/ URL.

Update timing recommendations

Clients should send a version update at every start up. Long running processes should send an update at least once every 24 hours, but not more than every hour.

Example client in Python

<TODO>

REST API details

All response bodies are JSON objects.

The examples below assume Partisci is running on localhost, port 7777 (default).

GET app/

The response contains a distinct list of all known application names, app_id, and last_update for any version of the app from any host.

$ curl 'http://localhost:7777/api/v1/app/' | python -m json.tool
{
    "data": [
        {
            "app": "Demo App B", 
            "app_id": "demo_app_b", 
            "host_count": 2, 
            "last_update": 1331137064
        }, 
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host_count": 2, 
            "last_update": 1331137064
        }
    ]
}

GET host/

The response contains a distinct list of all known hosts and last_update for any version and any application.

$ curl 'http://localhost:7777/api/v1/host/' | python -m json.tool
{
    "data": [
        {
            "app_count": 2, 
            "host": "host2.example.com", 
            "last_update": 1331137064
        }, 
        {
            "app_count": 2, 
            "host": "host1.example.com", 
            "last_update": 1331137064
        }
    ]
}

GET version/

The response contains every app_id, host, ver combination known. Only the most recent version is saved for every app_id, host pair.

$ curl 'http://localhost:7777/api/v1/version/' | python -m json.tool
{
    "data": [
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }, 
        {
            "app": "Demo App B", 
            "app_id": "demo_app_b", 
            "host": "host2.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "2.0"
        }, 
        {
            "app": "Demo App B", 
            "app_id": "demo_app_b", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }, 
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host2.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "2.0"
        }
    ]
}

GET version/?app=A

app_id can be used as a parameter to filter the results.

$ curl 'http://localhost:7777/api/v1/version/?app_id=demo_app_a' | python -m json.tool
{
    "data": [
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }, 
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host2.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "2.0"
        }
    ]
}

GET version/?app=A&ver=V

ver can be added to see a specific app / ver combination. Useful to see which hosts are running a version which needs updating.

$ curl 'http://localhost:7777/api/v1/version/?app_id=demo_app_a&ver=1.0' | python -m json.tool
{
    "data": [
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }
    ]
}

GET version/?host=H

host can be used as a parameter to filter the results. Either alone inventory all applications:

$ curl 'http://localhost:7777/api/v1/version/?host=host1.example.com' | python -m json.tool
{
    "data": [
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }, 
        {
            "app": "Demo App B", 
            "app_id": "demo_app_b", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }
    ]
}

or for a specific application:

$ curl 'http://localhost:7777/api/v1/version/?app_id=demo_app_a&host=host1.example.com' | python -m json.tool
{
    "data": [
        {
            "app": "Demo App A", 
            "app_id": "demo_app_a", 
            "host": "host1.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137064, 
            "ver": "1.0"
        }
    ]
}

POST update/

Clients can POST a version update body to this url.

$ curl 'http://localhost:7777/api/v1/update/' --data '{"instance": 0, "host": "terminal.example.com", "ver": "1.0", "app": "updatenator"}'
$ curl 'http://localhost:7777/api/v1/version/?app_id=updatenator' | python -m json.tool
{
    "data": [
        {
            "app": "updatenator", 
            "app_id": "updatenator", 
            "host": "terminal.example.com", 
            "host_ip": "::1", 
            "instance": 0, 
            "last_update": 1331137066, 
            "ver": "1.0"
        }
    ]
}

Error results

All error results will be returned with an appropriate HTTP status code and a JSON document body in the following format:

{
    "error" : "<ERROR MESSAGE>"
}

For example, an update missing keys: (curl -i also includes response headers in the output)

$ curl -i 'http://localhost:7777/api/v1/update/' --data '{}'
HTTP/1.1 415 Unsupported Media Type
Date: Wed, 07 Mar 2012 16:17:47 GMT
Transfer-Encoding: chunked
Content-Type: text/plain; charset=utf-8

{"error":"ERROR: ApiUpdate: parsePacket:\n  value for app & ver must be specified"}

Or attempting a GET on a POST only resource:

$ curl -i 'http://localhost:7777/api/v1/update/'
HTTP/1.1 405 Method Not Allowed
Date: Wed, 07 Mar 2012 16:17:47 GMT
Transfer-Encoding: chunked
Content-Type: text/plain; charset=utf-8

{"error":"ERROR: ApiUpdate: only accepts POST requests"}

Testing

Requirements & running the tests

Partisci is written in Go (golang.org).

Unit and benchmark tests are written using the Go testing package and can be run from the root of the Partisci repository:

go test ./...

The REST API test suite is written in Python (python.org). Requirements:

  • partiscid built and available on your path
  • Python 2.7
  • py.test (pytest.org)
  • requests (python-requests.org)

Open a shell to the test directory and run:

$ py.test

Implementation Status

Implemented

  • Initial documentation
  • Python update module
  • UDP listener
  • in-memory Version storage
  • sqlite backed Version storage (-sqlite)
  • REST API
  • command line update client (and golang client API)
  • optionally only keep recent updates (-trim)

Planned - soon

  • document API error results
  • add API support for returning a single AppSummary or HostSummary
  • rename? verstat? verstate? verinuse? VerInUse? verusage?
  • golang post about ok, err combination
  • experiment with cross-compiling windows/linux
  • Setup build system
    • create source distribution package with pre-built documentation
    • Windows binaries
    • OS/X binaries
    • linux binaries
  • write quickstart documentation
  • profile update loop (pass pointer to V instead of V?)
  • profile high update load, add buffer to updates chan?
  • profile Version queries
  • create partisci_fuzz tool to synthesize many fake updates
  • test server with MAXGOPROCS > 1

Planned - later

  • add commands to partisci client for most API calls
  • rename python module pypartisci –> partisci
  • PYPI package for python update module.

Possible

  • include relative URLs to queries in API results
    • overview –> summary
    • summary –> versions/&with?parameters
  • add overview API
  • implement other persistence options: redis, etc
  • python client: add start_update_thread()
  • add app and host version summaries with counts of each version
  • gzip responses when possible
  • Add paging to REST results
  • add since query parameter, which only returns newer version entries
  • store and return app_data with each Version, allowing custom application data
  • Store more historical data from updates.
    • Last update time for each app/version/machine. This would give a full version history for each machine.

Indices and tables