Pull 'n' Push

Pulls data from sources and pushes it to sinks with optional transformations in between.

Installation

pip install pnp

Installation with extras:

pip install pnp[fswatcher,faceR]

Please consult the component documentation to see if a component requires an extra or not.

Getting started

Define pulls to fetch / pull data from source systems. Define one push or multiple pushes per pull to transfer the pulled data anywhere else (you only need a plugin that knows how to handle the target). You configure your pipeline in yaml:

tasks:
  - name: hello-world
    pull:
      plugin: pnp.plugins.pull.simple.Repeat
      args:
        interval: 1s
        repeat: "Hello World"
    push:
      - plugin: pnp.plugins.push.simple.Echo

Copy this configuration and create the file helloworld.yaml. Run it:

pnp helloworld.yaml

This example yields the string Hello World every second.

Hint: You can validate your config without actually executing it with

   pnp --check helloworld.yaml

If you want to learn more please see the documentation at Read the Docs.