Commit Graph

145 Commits

Author SHA1 Message Date
Jason Schwarzenberger
9fff1b9e46 avoid duplicate articles listed on the category page 2020-11-04 17:14:42 +13:00
Jason Schwarzenberger
16b59f6c67 try stop bad pages. 2020-11-04 16:34:31 +13:00
Jason Schwarzenberger
939f4775a7 better settings example. 2020-11-04 15:52:34 +13:00
Jason Schwarzenberger
9bfc6fc6fa scraper settings, ordering and loop. 2020-11-04 15:47:12 +13:00
Jason Schwarzenberger
6ea9844d00 remove useless try blocks. 2020-11-04 15:37:19 +13:00
Jason Schwarzenberger
1318259d3d imply referrer is substack. 2020-11-04 15:21:07 +13:00
Jason Schwarzenberger
98a0c2257c increase declutter timeout. 2020-11-04 15:15:00 +13:00
Jason Schwarzenberger
e6976db25d fix tabs 2020-11-04 15:04:20 +13:00
Jason Schwarzenberger
9edc8b7cca move scraping for article content to files. 2020-11-04 15:00:58 +13:00
Jason Schwarzenberger
d718d05a04 fix dates for newsroom. 2020-11-04 11:53:16 +13:00
Jason Schwarzenberger
9f4ff4acf0 remove unnecessary sitemap.xml request. 2020-11-04 11:22:15 +13:00
Jason Schwarzenberger
db6aad84ec fix mistake. 2020-11-04 11:12:01 +13:00
Jason Schwarzenberger
29f8a8b8cc add news site categories feed. 2020-11-04 11:08:50 +13:00
9a279d44b1 Add header to get content type 2020-11-03 20:27:43 +00:00
Jason
abf8589e02 fix sitemap 2020-11-03 10:53:40 +00:00
Jason
b759f46582 use extruct for opengraph/json-ld/microdata of articles 2020-11-03 10:31:36 +00:00
Jason Schwarzenberger
736cdc8576 fix mistake. 2020-11-03 17:04:46 +13:00
Jason Schwarzenberger
244d416f6e settings config of sitemap/substack publications. 2020-11-03 17:01:29 +13:00
Jason Schwarzenberger
5f98a2e76a Merge remote-tracking branch 'tanner/master' into master
And adding relevant setings.py.example/etc.
2020-11-03 16:44:02 +13:00
Jason Schwarzenberger
76f1d57702 sitemap based feed. 2020-11-03 16:00:03 +13:00
Jason Schwarzenberger
4e64cf682a add the bulletin. 2020-11-03 12:41:16 +13:00
Jason Schwarzenberger
c5fe5d25a0 add substack.py top sites, replacing webworm.py 2020-11-03 12:28:39 +13:00
Jason
283a2b1545 fix webworm comments 2020-11-02 22:06:43 +00:00
Jason Schwarzenberger
0d6a86ace2 fix webworm dates. 2020-11-03 10:31:14 +13:00
Jason Schwarzenberger
f23bf628e0 add webworm/substack as a feed. 2020-11-02 17:09:59 +13:00
ca78a6d7a9 Move feed and Praw config to settings.py 2020-11-02 02:26:54 +00:00
e59acefda9 Remove Whoosh 2020-11-02 00:22:40 +00:00
cbc802b7e9 Try Hackernews API twice 2020-11-02 00:17:22 +00:00
4579dfce00 Improve logging 2020-11-02 00:13:43 +00:00
feba8b7aa0 Make qotnews work with WaPo 2020-10-29 04:55:34 +00:00
992c1c1233 Monkeypatch earlier 2020-10-24 22:30:00 +00:00
88d2216627 Add a script to delete a story 2020-10-03 23:42:21 +00:00
6cf2f01b08 Adjust feeds 2020-10-03 23:41:57 +00:00
6576eb1bac Adjust content-type request timeout 2020-08-14 03:57:43 +00:00
472af76d1a Adjust port 2020-08-14 03:57:18 +00:00
4727d34eb6 Delete displayed-attributes when init search 2020-08-14 03:56:47 +00:00
0e086b60b8 Remove business subreddit from feed 2020-08-14 03:55:28 +00:00
b46ce36c63 Update requirements 2020-07-08 05:24:32 +00:00
9a449bf3ca Remove extra logging 2020-07-08 02:36:40 +00:00
0bd9f05250 Fix crash when HN feed fails 2020-07-08 02:36:40 +00:00
9c116bde4a Remove document img and ignore r/technology 2020-07-08 02:36:40 +00:00
ebedaef00b Tune search rankings and attributes 2020-07-08 02:36:40 +00:00
d7f0643bd7 Add more logging 2020-07-08 02:36:40 +00:00
f1c846acd0 Remove get first image 2020-07-08 02:36:40 +00:00
850b30e353 Add requests timeouts and temporary logging 2020-07-08 02:36:40 +00:00
d614ad0743 Integrate with external MeiliSearch server 2020-07-08 02:36:40 +00:00
f46cafdc90 Integrate sqlite database with server 2020-07-08 02:36:40 +00:00
873dc44cb1 Update whoosh migration script 2020-07-08 02:36:40 +00:00
1fb9db3f4b Store ref list in database too 2020-07-08 02:36:40 +00:00
b923908a45 Begin initial sqlite conversion 2020-07-08 02:36:40 +00:00
dbdcfaa921 Check if cache is broken 2020-07-08 02:36:40 +00:00
8799b10525 Fall back to ref on manual submission title 2020-07-08 02:36:40 +00:00
6430fe5e9f Check content-type 2020-07-08 02:36:40 +00:00
a4cf719cb8 Remove technology subreddit 2020-07-08 02:36:40 +00:00
595f469b4a Update tildes parser group tag 2020-07-08 02:36:40 +00:00
7b31fcf690 Remove keys of uncached stories 2020-01-28 04:20:05 +00:00
b3d2eeb67f Fix tildes deleted comment parser error 2020-01-28 04:19:26 +00:00
9078b567f0 Add del tag and sort tags 2020-01-04 23:37:41 +00:00
2822974b6e Stop using archive.is on articles (hits CAPTCHAs) 2019-12-15 22:47:33 +00:00
17ef7e3a65 Whitelist more html tags 2019-12-14 07:39:10 +00:00
2d80b19414 Grab comments on manually submitted links 2019-12-02 23:15:51 +00:00
ebcbf1b624 Sanitize html 2019-12-01 22:18:41 +00:00
e231cd5c31 Decrease feed cache length to 150 2019-12-01 22:18:14 +00:00
db5097ac57 Drop articles more than two days old 2019-11-08 21:50:33 +00:00
2edb3ceba7 Allow manual submission of articles 2019-11-08 05:55:30 +00:00
38b5f2dbeb Move to gevent production http server 2019-11-08 02:37:57 +00:00
6826f731c7 Handle hostnames better 2019-11-07 22:10:08 +00:00
bb693ba434 Add subreddit 2019-11-07 22:09:45 +00:00
9e55f6e4ec Fix Tildes down for maintenance edge case 2019-10-22 05:01:30 +00:00
edc4c439d7 Prefetch first images 2019-10-19 07:33:06 +00:00
f8998b687e Fix crash from domain and ext check bug 2019-10-16 08:56:31 +00:00
e4f81472fc Fix copy/paste error, switch to info logging 2019-10-16 05:26:47 +00:00
f293f2b5f9 Begin README and add license 2019-10-15 16:40:55 -06:00
810e8c5ead Archive WSJ articles first, catch KeyboardInterrupt 2019-10-15 21:03:47 +00:00
9c4766a928 Stop using python keyword id for id 2019-10-15 20:36:20 +00:00
0f5b2a5ff9 Cache all articles in IndexedDB 2019-10-12 23:41:31 +00:00
7cb87b59fe Move archive to Whoosh and add search 2019-10-12 05:32:17 +00:00
45b75b420b Gitkeep archive directory 2019-10-10 21:55:21 +00:00
f0721519e1 Serve client through apiserver, adding meta info 2019-10-10 21:54:29 +00:00
5fd4fdb21c Fix Tildes comments with unknown authors 2019-10-08 08:01:17 +00:00
19e9a80be1 Archive Bloomberg articles first 2019-10-08 08:00:50 +00:00
5caa4542d8 Gitkeep apiserver data directory 2019-10-08 07:59:30 +00:00
0053147226 Ignore certain files and domains, remove refs 2019-09-24 08:22:06 +00:00
0496fbba45 Ignore new Tildes posts and handle deleted ones 2019-09-24 08:21:26 +00:00
0a1ebaa8b8 Handle Reddit PRAW exceptions 2019-09-24 08:20:46 +00:00
2ede5ed6ff Filter out False comments 2019-08-30 06:23:14 +00:00
23cdbc9292 Render reddit markdown, poll tildes better, add utils 2019-08-28 04:13:02 +00:00
fc8ce79e33 Try outline.com for reader mode first 2019-08-25 23:49:08 +00:00
cf9e197e6c Fix tildes comments parsing bug 2019-08-25 07:46:22 +00:00
1b6c8fc6cb Add tildes to feeds 2019-08-25 00:36:26 +00:00
a2509958da Add reddit to feeds 2019-08-24 21:37:43 +00:00
d341d4422f Abstract api server feeds 2019-08-24 08:49:11 +00:00
c1a81a4d8c Write news stories to disk 2019-08-24 05:07:16 +00:00
62d68da415 Finish prototype api server 2019-08-23 08:23:48 +00:00
c04b5c27f2 Figure out .gitignores 2019-08-23 08:23:26 +00:00