9.9 KiB

Raw Permalink Blame History

TuckIt

This is a CLI tool with a focus on torrents and doing batches of things in i2p. Can handle torrent creation by pasting paths in, eephead many sites at once with a .txt file, batch resize/clean metada from all images found in a directory, and organize lots of port forwards and snark instances.

Main tools: torrent creation, multisnark, port forward manager, image resizing, and screenshots from video.

To Install:

Running this should work:

curl --retry 5 -x http://127.0.0.1:4444 -O http://git.simp.i2p/simp/TuckIt/raw/branch/main/installer.py && python3 installer.py

The installer will download dependencies in a virtual environment for you. Running with python3 tuckit will start it inside the virtual environment (this is the same as doing cd tuckit && python3 __main__.py)

If that doesn't work

Clone the repo (or download the zip and unzip if this doesn't work)

git clone --config http.proxy=127.0.0.1:4444 http://git.simp.i2p/simp/TuckIt.git

Go into the TuckIt directory and make a virtual environment there

python3 -m venv ./

Activate the environement by going into /bin and doing:

source activate

then install the requirements with

pip3 install -r requirements.txt

Notes

Fair warning this code is 99% ass

This is an early release and there are bugs. There's some bugginess with the CLI menus especially, but the core functions (many of which were made separately and mushed together here) are solid for the most part.

Many of these tools I regularly use so this is a project that is being actively developed. Most of my other tools will be moved to this one.

Major dependencies include beautifulsoup for html parsing, aiohttp for async http requests, moviepy for screenshots, pillow for resizing images, and torf for torrent file manipulation and hashing.

Multisnark tool info

This has been mostly tested on snark+ standalone. It should work on snark standalones 2.4 and onwards and has been tested to work on 2.8. Embedded should also work as it's the same.

This relies on parsing the html and being able to reach the snark (hence the port forward manager for dealing with remote machines), so changes to the snark interface can break it. I've kept support for the older snarks that use a different style footer.

Right now there isn't a good way to get the live tunnel count from snark standalone. This may change in the future, it's assuming 16 tunnels in each direction at all times for stats. When not being pushed, snark will use less tunnels down to 2/2 in each direction when idle, so average tunnel pressure metric won't be accurate otherwise.

Features

Torrent

Torrent file creation: Handle batches by adding each path before hashing. The default uses torf and sets creation time to UTC with some random time offset. Mktorrent can still be used by enabling it, but it's recommended to use torf. Some filtering to ignore hidden and other unwanted files.
Tracker uptime monitor: in the background it can ping the trackers you've added.
Torrent metainfo: View contents of a torrent file, metainfo, files.
Scrape trackers: Gets peer counts from trackers in your tracker list with the infohash.
Batch .torrent file editing (basic): Change trackers, comments, private flag. Giving a directory will batch edit .torrent files found. Tracker option will replace trackers with your tracker list so you can bring clearnet .torrents into i2p this way.

Multisnark

Overview: Basic overview with upload/download status for all your snarks in one place.
Start/stop all at once: When multiple snarks are sharing a directory this can be useful for uploading.
Single snark options: Add torrents, stop and start, adjust settings, copy settings from one start to another
Force outbound quantity > 16: Can allow for faster speeds if connecting to a lot of peers

Utilities

ssh portforward manager: Auto connects and maintains port forwards. Can group ports you want connected together per host separately. Handy to manage several external routers and snark instances. The config file is hopefully easy to setup, so it's also good for easily dealing with lots of port forwards.
eephead: Get headers using i2p http proxy along with ping. Accepts list of sites to check or .txt file with 1 eepsite per line.
eepget: Downloads a file or list of files into a downloads directory.
todolist Very simple todolist for convenience sake.
notbob: This is just a shortcut for notbobs cli tool that shows new eepsites.

Images

Batch image resizing: Erases metadata of images and batch resize postman or other uses. If giving a directory, it will resize all images found.
Screenshot maker: Grabs screenshots from a video source, or accepts a directory to do all videos found. Settings for producing collage and/or raw screenshots.

Development direction

There's room for performance boosts. Right now this uses a combination of async and threading for different tasks. In most cases threads should stay under 5, with memory usage around 50mb. Major bottlenecks are:

Startup - the order of loading dependencies can be optimized to make this faster

Screenshots - multiprocessing could be used if we want to make this faster. Personally, i don't think this is a huge issue

Multisnark - the real bottleneck here is likely on loading the html page and not parsing in the script. i'm pulling from /i2psnark/.ajax/xhr1.html?filter=active so the html should be minimal, and aiohttp is used for async requests. Commands are still sent out seqentially

Port forward - this runs in its own thread sequentially. it could be made to run async or as a daemon with this script getting status info from it.

Other functions that will be added:

load list of trackers from external site
optimize for async, reduce threading that's not needed
use of sam for various functions, especially ones where transient b32 is useful
metadata from external database
mediainfo
metadata insertion into .torrent files, support for api
port forward script and other background tasks offloaded to a daemon

Wishlist, longer term:

Full automated .torrent upload
Integration of tokenized file transfer script
Integrated drop in personal torrent indexer
adapt irc bots using sam to cli irc client
deadman switch and/or encryption

Config setup

On the first run it will ask to make a config.ini file with default values. Enter 'y' to do this and restart the script.

To change these do -c in the script. All values in the config file can be changed through the script with the exception of the port_forward options. Here's what the options do:

General settings

[settings]
ping_frequency = 21600    >time to check tracker alive status (seconds)
ping_trackers = False    >ping trackers in background while script is running? (T/F)
auto_ssh = False    >Attempt to forward SSH ports in background while running? > (T/F)
use_mktorrent = False    >If False, torf is used. If True, a command is given to mktorrent, so you must have mktorrent installed. Recommended to leave this as is.

Proxies

One option for now just using the http proxy of your i2p router. Likely will include other options later dealing with clearnet apis

[proxies]
i2p_router = 127.0.0.1:4444

Port forward

You can have as many as you want of these for grouping and organizing. The local_port and remot_port are lists, where each item in the list is matched with the other in the same order. This was done mainly to simplify adding new ports. You can have separate blocks using the same host. So this can be used to separate ports you want always forwarded and ports you want to forward manually when you need them.

[port_forward:hostname]    >'hostname' is an arbitrary name to group your connections by.
local_port = 4444, 7656, 7654    >local port you want to access the remote port from.
remote_port = 4444, 7656, 7654    >remote port you're forwarding
localhost = 127.0.0.1    >localhost, usually leave this as is, but you can also do different values here if needed.
user = user    >user for login to remote host
remote_host = 123.123.123.123    >remote host ip address
ssh_port =    >if blank, port 22 will be used.
auto_start = t    >if auto_ssh is T, do you want this auto-connect? this way you can have manual groupings.

Tracker

List of trackers. The first tracker in the list is the primary one.

Images

includes default target size for resizing images and save directories. By default resized images will go to /resized

[images]
img_target_size = 100 > default target size for image resize (kb)

[image_directories]
img_default_directories = /path/1 > the first value needs to be after the "=" on the same line, then one per line.
/path/2

Todo

This is just a simple todo list, add/delete items in the script.

[todo_list]
todo = some todo list things

Snarks

One per line, snark1/snark2 being arbitrary names. Dictionary values of url (host:port of snark instance) and if it's embedded (t/f). 'embedded' is actually if it should try to connect with tls. Embedded snark is reffering to if the snark is part of the router console or not. If it's snark standalone, use False.

[snarks]
snark1 = {'url': '127.0.0.1:8001', 'embedded': False}
snark2 = {'url': '127.0.0.1:8002', 'embedded': False}

snark_settings

Because settings are sent to snark with a post command, it checks to see the previous settings before sending them out. Since > 16 for tunnel quantity won't show up right on retrieval, if we want to use > 16 we need to override it.

If you set this > 0 it will always send the outbound quantity from this setting.

[snark_settings]
force_outbound_quantity = 32

sam

This is unnused for now.

[sam]
sam_host_port = 127.0.0.1:7654

Status

This is just for storing the status of trackers. There shouldn't be anything here worth editing.

9.9 KiB Raw Permalink Blame History