Commit Graph

161 Commits

Author SHA1 Message Date
Jason Schwarzenberger
4e5dc65461 don't rescrape if simple. 2020-12-10 16:25:51 +13:00
Jason Schwarzenberger
33a25fa34e allow re-scraping if simple scraper was used. 2020-12-04 15:34:04 +13:00
Jason Schwarzenberger
da7f6330bf improve meta data scraping. 2020-12-04 12:46:46 +13:00
Jason Schwarzenberger
2a2bf4d671 add excerpt and scraper details. 2020-12-03 16:41:27 +13:00
Jason
afe3e08055 etc 2020-12-03 01:28:10 +00:00
Jason Schwarzenberger
d1c513b9d6 move purify to server side. 2020-12-02 15:46:06 +13:00
Jason Schwarzenberger
e53c5fc904 fix mistake. 2020-12-02 13:28:08 +13:00
Jason Schwarzenberger
59c6f17e67 gotta try this on live. 2020-12-02 13:22:47 +13:00
Jason Schwarzenberger
f670479bd7 progress 2020-11-30 18:11:45 +13:00
Jason
085dd47d13 fix tvnz time for nzst/nzdt 2020-11-26 00:35:32 +00:00
Jason Schwarzenberger
72e2232469 fix substack comments. 2020-11-26 11:17:25 +13:00
Jason Schwarzenberger
247715a76e adjust feed thread. 2020-11-25 12:34:46 +13:00
Jason Schwarzenberger
5c96092a57 sort ref_list so newly added is first. 2020-11-24 17:28:24 +13:00
Jason Schwarzenberger
bb1413b586 sort substack feed by time. 2020-11-24 10:56:38 +13:00
Jason Schwarzenberger
fe01ea52e5 get favicons for custom substack publications. 2020-11-24 10:36:31 +13:00
Jason Schwarzenberger
3daae5fa1b change substack time parsing to misc.time 2020-11-23 16:46:54 +13:00
Jason Schwarzenberger
c1b6349771 namespace the refs for hn and substack. 2020-11-23 16:09:12 +13:00
Jason
54a4c7e55a fix with try-catch 2020-11-23 01:20:40 +00:00
Jason
b12a3570b0 add logging, extend id length 2020-11-21 21:21:31 +00:00
Jason Schwarzenberger
a86eb98c1a fix hn self posts related discussion. 2020-11-20 13:06:19 +13:00
Jason Schwarzenberger
abf7f0a802 force reader update in update-story.py 2020-11-20 12:21:27 +13:00
Jason Schwarzenberger
32bc3b906b add update-story.py 2020-11-19 15:06:55 +13:00
Jason Schwarzenberger
f5e65632b8 fix comment date. 2020-11-19 14:27:24 +13:00
Jason Schwarzenberger
1fe524207e stuff comments. 2020-11-19 14:23:01 +13:00
Jason Schwarzenberger
539350a83d port separation. 2020-11-18 17:21:37 +13:00
Jason Schwarzenberger
f5b38f5c6b remove readerserver, add declutter. 2020-11-18 12:59:35 +13:00
Jason Schwarzenberger
3b885e4327 renaming things. 2020-11-17 15:54:14 +13:00
Jason Schwarzenberger
5668fa5dbc fix mistake. 2020-11-17 12:54:54 +13:00
Jason Schwarzenberger
b771b52501 add regex to get a unique ref from each sitemap/category based article url. 2020-11-17 12:38:28 +13:00
Jason Schwarzenberger
f5ccd844da fix import error. 2020-11-16 15:41:09 +13:00
Jason Schwarzenberger
6a91b9402f split categories, sitemap and other crap out of news.py 2020-11-16 15:30:33 +13:00
Jason Schwarzenberger
b23e470317 move reddit thresholds as settings variables. 2020-11-16 10:11:39 +13:00
Jason Schwarzenberger
7420b5ece9 fix microdata multiple authors 2020-11-12 17:33:46 +13:00
Jason Schwarzenberger
64ced635cc fix mistake. 2020-11-12 17:15:29 +13:00
Jason Schwarzenberger
9318627f1b ability to pass in multiple site maps/category urls. 2020-11-12 17:11:51 +13:00
Jason Schwarzenberger
3d0a3f1577 support list based json-ld authors. 2020-11-12 15:08:23 +13:00
Jason Schwarzenberger
587b10c438 recursive sitemaps (sitemap indexes) 2020-11-12 14:56:46 +13:00
Jason
00954c6cac local browser scraper 2020-11-11 09:26:54 +00:00
Jason Schwarzenberger
3169af3002 hostname from settings. 2020-11-11 09:46:27 +13:00
Jason Schwarzenberger
d588a60930 add source to searchable attributes. 2020-11-11 09:37:54 +13:00
Jason Schwarzenberger
408e2870b2 tzinfo and microdata schema urls. 2020-11-10 16:51:27 +13:00
Jason Schwarzenberger
44b8b36547 add data cast in query. 2020-11-10 15:50:18 +13:00
Jason Schwarzenberger
1d78b1c592 fix favicon url. 2020-11-10 15:34:21 +13:00
Jason Schwarzenberger
0374794536 Sitemap and Category to get favicon into icon property of story. 2020-11-10 15:22:27 +13:00
Jason Schwarzenberger
5efc6ef2d3 add related stories (in api only) 2020-11-10 14:09:56 +13:00
Jason Schwarzenberger
4ec50e20cb feed thread loop. 2020-11-10 10:10:38 +13:00
Jason Schwarzenberger
c1b7877f4b remove limit. 2020-11-09 17:54:50 +13:00
Jason Schwarzenberger
7b8cbfc9b9 try to make feed only determined by the max age. 2020-11-09 17:50:58 +13:00
Jason Schwarzenberger
bfa4108a8e Merge remote-tracking branch 'tanner/master' 2020-11-09 16:08:28 +13:00
Jason Schwarzenberger
0bd0d40a31 use json type in sqlite. 2020-11-09 15:45:10 +13:00