The Viper Python API¶

The Concept¶

Viper provides a powerful collection of data types such as Hosts, Runners, Results etc. and uses method chaining to perform different operations. The viper.collections module contains the collection of such data types. These data types share some common properties as all they inherit from the Collection class.

Example: Method Chaining¶

from viper import Hosts
import task

print(
    Hosts.from_file("hosts.csv")
    .task(task.ping())
    .run(max_workers=50)
    .final()
    .order_by("host.hostname", "host.ip")
    .to_file("results.csv")
    .format("{host.hostname}: {stdout}")
)

Tip

Refer to Getting Started to see how task.ping and hosts.csv are written.

Unit vs Container Types¶

The above mentioned data types can be categorised as unit and container types. The unit ones inherit from the Item class, while the container types inherit from Items class.

Below are the list of unit types and their container type counterparts:

Unit Types	Container Types
`Task`
`Host`	`Hosts`
`Runner`	`Runners`
`Result`	`Results`

Useful Common Properties & Abilities¶

The properties mentioned below are common to both unit and container type objects.

Immutable: All the datatypes are immutable i.e. they cannot be modified once initialized. This is to prevent any unexpected behaviour caused due to stateful-ness.
.from_json() and .to_json(): All the objects can be initialized from JSON texts using the .from_json() factory method and can be dumped back to JSON using the .to_json() method. This enables the objects to use a wide range of mediums such as the Unix pipes.
.format(): The objects can be converted to a string with a custof format using the .format() method.

Example:
```
host.format("{ip} {hostname} {meta.tag}")
```

Useful Abilities Common to the Unit Types¶

These abilities are common to Task, Host, Runner and Result unit type objects.

.from_dict() and .to_dict(): Helps representing the objects as Python dictionaries.

Useful Abilities Common to the Container Types¶

These abilities are common to Hosts, Runners and Results container type objects.

.from_items() and .to_items(): The .from_items() factory method is the recommended way to initialize container type objects. Although it can be a little slower, it removes duplicate items and performs other important checks before initializing the object. It supports sequences, generators, unit objects or all at once.
Attention
```
# Bad
Hosts((host1, host2, host3))

# Good
Hosts.from_items(host1, host2, host3)
```
The .to_items() or the alias .all() returns the tuple of unit items back.

Example:
```
Hosts.from_items(
    host1, host2                      # Unit objects
    [host3, host4],                   # Sequence of objects
    (host for host in list_of_hosts)  # Generator of objects
).to_items()
```
.from_file() and .to_file(): Container type objects can be initialized from text files and dumped back to text files with certain formats (currently supported json, yml and csv) using these methods.

Example:
```
Hosts.from_file("hosts.json").to_file("hosts.csv")
```
.from_list() and .to_list(): Similar to unit types’ .from_dict() and .to_dict() but operates with list of dictionaries that represent the unit type objects.
.count(): Returns the count of items it holds.
.head() and .tail(): Returns an instance of the same container type object containing first or last n items (n defaults to 10).

Example:
```
# Get the set of last 5 items from the set of first 10 items.
hosts.head(10).tail(5)
```
.range(): Similar to .head() or .tail() but enables us to define a range (like Python’s list[i:j] indexing).

Example:
```
# Exclude the last item (like like Python's list[0:-1])
hosts.range(0, -1)
```
.sort(): Similar to Python’s list.sort() but returns a new instance instead of making changes to the existing object (which is impossible because of immutability).

Example:
```
# Reverse sort by IP, then by hostname
hosts.sort(key=lambda host: [host.ip, host.hostname], reverse=True)
```
.order_by(): Similar to .sort() but expects the field names instead of a function. Inspired by SQL.

Example:
```
# Reverse sort by ip, then by hostname
hosts.order_by("ip", "hostname", reverse=True)
```
.filter(): Similar to Python’s filter() but returns an instance of the same container type object containing the filtered items.

Example:
```
# Filter hosts where hostname starts with "foo"
hosts.filter(lambda host: host.hostname.startswith("foo"))
```
.where(): Similar to filter, but expects the field name, the condition and the value instead of a function. Inspired by SQL.

Example:
```
# Filter hosts where the hostname starts with "foo"
hosts.where(
    "hostname", WhereConditions.startswith, ["foo"]
)
```

More on Task: Command Factories, Output Processors, Callbacks and …¶

The minimum requirements of defining a Task is to pass the task name and the command factory. Optionally, we can also pass the stdout and stderr processors, and also the pre and post run callbacks.

The command factory expects a Host object and returns a tuple of string.

Example:

def ping_command(host):
    return "ping", "-c", "1", host.ip

The stdout and stderr processors expect a string and return a string.

Example:

def strip_output(txt):
    return txt.strip()

The pre run callback expects a Runner object and doesn’t return anything. While the post run callback expects a Result object and doesn’t return anything either.

Example:

import sys

def log_command_pre_run(runner):
    command = runner.task.command_factory(runner.host, *runner.args)
    print("Running command:", command, file=sys.stderr)

def log_result_post_run(result):
    print("OK:" if result.ok() else "ERROR:", result.host.hostname, file=sys.stderr)

Note

Logs are being printed to stderr as stdout is for the JSON encoded Results object.

Attention

The arguments command_factory, stdout_processor, stderr_processor, pre_run and post_run callbacks expect normal functions, not lambdas.

# Bad
def ping():
    return Task(
        name="Ping once",
        command_factory=lambda host: "ping", "-c", "1", host.ip,
        stdout_processor=lambda txt: txt.strip(),
        stderr_processor=lambda txt: txt.strip(),
        pre_run=lambda runner: print(runner.to_dict(), file=sys.stderr),
        post_run=lambda result: print(result.to_dict(), file=sys.stderr),
    )

# Good
def ping():
    return Task(
        name="Ping once",
        command_factory=ping_command,
        stdout_processor=strip_output,
        stderr_processor=strip_output,
        pre_run=log_command_pre_run,
        post_run=log_result_post_run,
    )

Apart from these, a Task also optionally expects timeout, retry and meta.

timeout: The execution will timeout after the specified seconds if timeout is defined.

The countdown doesn’t count the time spent on the pre and post run callbacks, neither the command factory invocation. It only counts time spent on executing the generated command.
retry: It defaults to 0. If more than 0, The runner will re-invoke the run() method with the updated retry value if the command execution fails. The results generated for these retries will be stored in DB and will be available in history. They will have the same trigger_time but different start and end time values.

However, if the failure is caused by any reason other than the actual command execution, such as while invoking the command factory or output processors or pre/post run callbacks, a Python error will be raised which won’t be stored in DB. If any such error occurs while running the task in batch, it will be ignored with the traceback printed to stderr.

meta: It is the same as the meta field in Host. The value should be generated only using the viper.meta() function.

Attention

# Bad
def ping():
    return Task(
        name="Ping once",
        command_factory=ping_command,
        meta={"tag": "foo"},
    )

# Good
def ping():
    return Task(
        name="Ping once",
        command_factory=ping_command,
        meta=meta(tag="foo")
    )

The Viper Python API¶

The Concept¶

Example: Method Chaining¶

Unit vs Container Types¶

Useful Common Properties & Abilities¶

Useful Abilities Common to the Unit Types¶

Useful Abilities Common to the Container Types¶

More on Task: Command Factories, Output Processors, Callbacks and …¶

Viper Infrastructure Commander

Navigation

Related Topics