161 Commits

Author	SHA1	Message	Date
Jason Schwarzenberger	4e5dc65461	don't rescrape if simple.	2020-12-10 16:25:51 +13:00
Jason Schwarzenberger	33a25fa34e	allow re-scraping if simple scraper was used.	2020-12-04 15:34:04 +13:00
Jason Schwarzenberger	da7f6330bf	improve meta data scraping.	2020-12-04 12:46:46 +13:00
Jason Schwarzenberger	2a2bf4d671	add excerpt and scraper details.	2020-12-03 16:41:27 +13:00
Jason	afe3e08055	etc	2020-12-03 01:28:10 +00:00
Jason Schwarzenberger	d1c513b9d6	move purify to server side.	2020-12-02 15:46:06 +13:00
Jason Schwarzenberger	e53c5fc904	fix mistake.	2020-12-02 13:28:08 +13:00
Jason Schwarzenberger	59c6f17e67	gotta try this on live.	2020-12-02 13:22:47 +13:00
Jason Schwarzenberger	f670479bd7	progress	2020-11-30 18:11:45 +13:00
Jason	085dd47d13	fix tvnz time for nzst/nzdt	2020-11-26 00:35:32 +00:00
Jason Schwarzenberger	72e2232469	fix substack comments.	2020-11-26 11:17:25 +13:00
Jason Schwarzenberger	247715a76e	adjust feed thread.	2020-11-25 12:34:46 +13:00
Jason Schwarzenberger	5c96092a57	sort ref_list so newly added is first.	2020-11-24 17:28:24 +13:00
Jason Schwarzenberger	bb1413b586	sort substack feed by time.	2020-11-24 10:56:38 +13:00
Jason Schwarzenberger	fe01ea52e5	get favicons for custom substack publications.	2020-11-24 10:36:31 +13:00
Jason Schwarzenberger	3daae5fa1b	change substack time parsing to misc.time	2020-11-23 16:46:54 +13:00
Jason Schwarzenberger	c1b6349771	namespace the refs for hn and substack.	2020-11-23 16:09:12 +13:00
Jason	54a4c7e55a	fix with try-catch	2020-11-23 01:20:40 +00:00
Jason	b12a3570b0	add logging, extend id length	2020-11-21 21:21:31 +00:00
Jason Schwarzenberger	a86eb98c1a	fix hn self posts related discussion.	2020-11-20 13:06:19 +13:00
Jason Schwarzenberger	abf7f0a802	force reader update in update-story.py	2020-11-20 12:21:27 +13:00
Jason Schwarzenberger	32bc3b906b	add update-story.py	2020-11-19 15:06:55 +13:00
Jason Schwarzenberger	f5e65632b8	fix comment date.	2020-11-19 14:27:24 +13:00
Jason Schwarzenberger	1fe524207e	stuff comments.	2020-11-19 14:23:01 +13:00
Jason Schwarzenberger	539350a83d	port separation.	2020-11-18 17:21:37 +13:00
Jason Schwarzenberger	f5b38f5c6b	remove readerserver, add declutter.	2020-11-18 12:59:35 +13:00
Jason Schwarzenberger	3b885e4327	renaming things.	2020-11-17 15:54:14 +13:00
Jason Schwarzenberger	5668fa5dbc	fix mistake.	2020-11-17 12:54:54 +13:00
Jason Schwarzenberger	b771b52501	add regex to get a unique ref from each sitemap/category based article url.	2020-11-17 12:38:28 +13:00
Jason Schwarzenberger	f5ccd844da	fix import error.	2020-11-16 15:41:09 +13:00
Jason Schwarzenberger	6a91b9402f	split categories, sitemap and other crap out of news.py	2020-11-16 15:30:33 +13:00
Jason Schwarzenberger	b23e470317	move reddit thresholds as settings variables.	2020-11-16 10:11:39 +13:00
Jason Schwarzenberger	7420b5ece9	fix microdata multiple authors	2020-11-12 17:33:46 +13:00
Jason Schwarzenberger	64ced635cc	fix mistake.	2020-11-12 17:15:29 +13:00
Jason Schwarzenberger	9318627f1b	ability to pass in multiple site maps/category urls.	2020-11-12 17:11:51 +13:00
Jason Schwarzenberger	3d0a3f1577	support list based json-ld authors.	2020-11-12 15:08:23 +13:00
Jason Schwarzenberger	587b10c438	recursive sitemaps (sitemap indexes)	2020-11-12 14:56:46 +13:00
Jason	00954c6cac	local browser scraper	2020-11-11 09:26:54 +00:00
Jason Schwarzenberger	3169af3002	hostname from settings.	2020-11-11 09:46:27 +13:00
Jason Schwarzenberger	d588a60930	add source to searchable attributes.	2020-11-11 09:37:54 +13:00
Jason Schwarzenberger	408e2870b2	tzinfo and microdata schema urls.	2020-11-10 16:51:27 +13:00
Jason Schwarzenberger	44b8b36547	add data cast in query.	2020-11-10 15:50:18 +13:00
Jason Schwarzenberger	1d78b1c592	fix favicon url.	2020-11-10 15:34:21 +13:00
Jason Schwarzenberger	0374794536	Sitemap and Category to get favicon into icon property of story.	2020-11-10 15:22:27 +13:00
Jason Schwarzenberger	5efc6ef2d3	add related stories (in api only)	2020-11-10 14:09:56 +13:00
Jason Schwarzenberger	4ec50e20cb	feed thread loop.	2020-11-10 10:10:38 +13:00
Jason Schwarzenberger	c1b7877f4b	remove limit.	2020-11-09 17:54:50 +13:00
Jason Schwarzenberger	7b8cbfc9b9	try to make feed only determined by the max age.	2020-11-09 17:50:58 +13:00
Jason Schwarzenberger	bfa4108a8e	Merge remote-tracking branch 'tanner/master'	2020-11-09 16:08:28 +13:00
Jason Schwarzenberger	0bd0d40a31	use json type in sqlite.	2020-11-09 15:45:10 +13:00