I2pnews webapp and bot

This is a both the webapp at i2pnews.simp.i2p and the bot present in #i2p-news. It follows the feed list from rsslist.txt and periodically updates all feeds using aiohttp, asyncio and feedparser.

If a feed has been updated, it saves the last x amount of posts to a txt file under /static/rssfeed for each category. Each category is color-coded by the bot and given a seperate page and consolidated rss feed with FeedGenerator. Post times are changed to UTC.

The webapp displays a page that shows the ping times and last seen for each feed, giving a text list of alive feeds and all feeds given in json format at /json.

Todo

The bot was built first, and the webapp built around the irc bot. It'd be better if they were seperate with the webapp functioning independently and puppeting the ircbot through api, i think.

[DONE] Cap per feed per update: simple max of x number new posts on the irc bot so its not a wall of text. Feeds are capped at 5 in IRC for the live bot now.

[WIP] I2p urls: The feed links are all over the place, some point to clearnet, or localhost, or not the post. Have the links point to valid i2p only links. Localhost is corrected and some site specific corrections.

[X] Cleanup: a lot has been copied from other projects, it's messy

[X] Optimization: this could be leaner, it runs around 85MB now. If we keep the feeds in memory it can run faster, it's not doing that now.

Install

If for some reason you want to install this, that's probably a mistake on your part but, clone the repo using your i2p http proxy, cd into it, then make a python venv.

Activate the environment, then run pip3 install -r requirements.txt

Something like this should work:

git clone --config http.proxy=127.0.0.1:4444 http://git.simp.i2p/simp/i2pnews.git && cd i2pnews && python3 -m venv ./ && cd bin && source activate && cd .. && pip3 install -r requirements.txt

After requirements are installed, while in your venv running python3 wsgi.py should work, which will run gunicorn. Running python3 app.py will just run the flask app withou gunicorn. Useful for debugging not much else.

Recommended to run the wsgi.py through nginx or apache, example config with nginx pointing i2p tunnel to 5050:

server {
	listen 5050;
	server_name somedomain.i2p;
	server_tokens off;
	if ($request_method !~ ^(GET|POST)$ ) {
    	return 444;
	}
	if ($http_user_agent ~* LWP::Simple|BBBike|wget) {
    	return 444;
	}
	location / {
    	proxy_hide_header Server;
    	proxy_pass http://unix:/path/to/newsbot/newsbot.sock;
	}
}

Listens in on the socket on /path/to/newsbot/newsbot.sock so make sure you change that to the full directory.

Settings

Most settings are in website.ini file

[proxy]  
hostname = 127.0.0.1  
http_tunnel = 4444 - setup a seperate http tunnel under java to close after idling x minutes and open with new keys on activity  

[rss]  
news_bot = botname
news_pass = botpassword  
news_irc_port = 6667  
news_rss_flask_port = 5001 - port to serve flask. not used if using wsgi like gunicorn  
news_rss_ah = addresshelperlink  
news_email = nobody@mail.i2p  
auth_users = nobody, anothernick - irc nicks authed to do $admin commands  
channel = #i2p-news - original channel to join, changed with $admin main_channel #channel1 #channel2 or $admin join/part #channel  
primary = nobody - sends msgs to this nicks memoserv on admin commands  
gunicorn_port = 5000 - for running production with wsgi  

Add feeds

feed list is rsslist.txt, format is url category categories: news, blog, forum, developer, torrents example:

http://zzz.i2p/topics.rss forum

Then in IRC do:

$admin refresh