229 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Jason Schwarzenberger	587b10c438	recursive sitemaps (sitemap indexes)	2020-11-12 14:56:46 +13:00
Jason	00954c6cac	local browser scraper	2020-11-11 09:26:54 +00:00
Jason Schwarzenberger	637bc38476	fix mistake.	2020-11-11 17:21:31 +13:00
Jason Schwarzenberger	164b7e72c4	basically add declutter like capabilities.	2020-11-11 17:16:04 +13:00
Jason Schwarzenberger	3169af3002	hostname from settings.	2020-11-11 09:46:27 +13:00
Jason Schwarzenberger	d588a60930	add source to searchable attributes.	2020-11-11 09:37:54 +13:00
Jason Schwarzenberger	408e2870b2	tzinfo and microdata schema urls.	2020-11-10 16:51:27 +13:00
Jason Schwarzenberger	44b8b36547	add data cast in query.	2020-11-10 15:50:18 +13:00
Jason Schwarzenberger	4f49684194	remove logos from utils.js	2020-11-10 15:38:48 +13:00
Jason Schwarzenberger	1d78b1c592	fix favicon url.	2020-11-10 15:34:21 +13:00
Jason Schwarzenberger	0374794536	Sitemap and Category to get favicon into icon property of story.	2020-11-10 15:22:27 +13:00
Jason Schwarzenberger	943a1cfa4f	reader server	2020-11-10 14:56:21 +13:00
Jason Schwarzenberger	9cee370a25	tvnz icon	2020-11-10 14:10:02 +13:00
Jason Schwarzenberger	5efc6ef2d3	add related stories (in api only)	2020-11-10 14:09:56 +13:00
Jason Schwarzenberger	4ec50e20cb	feed thread loop.	2020-11-10 10:10:38 +13:00
Jason Schwarzenberger	c1b7877f4b	remove limit.	2020-11-09 17:54:50 +13:00
Jason Schwarzenberger	7b8cbfc9b9	try to make feed only determined by the max age.	2020-11-09 17:50:58 +13:00
Jason Schwarzenberger	bfa4108a8e	Merge remote-tracking branch 'tanner/master'	2020-11-09 16:08:28 +13:00
Jason Schwarzenberger	0bd0d40a31	use json type in sqlite.	2020-11-09 15:45:10 +13:00
Jason Schwarzenberger	4e04595415	fix search.	2020-11-09 15:44:44 +13:00
Jason	006db2960c	change to 3 days	2020-11-09 01:36:51 +00:00
Jason Schwarzenberger	1f063f0dac	undo log level change	2020-11-06 11:20:34 +13:00
Jason Schwarzenberger	1658346aa9	fix news.py feed.	2020-11-06 10:37:43 +13:00
Jason Schwarzenberger	2dbc702b40	switch to python-dateutil for parser, reverse sort xml feeds.	2020-11-06 10:02:39 +13:00
Jason Schwarzenberger	1c4764e67d	sort sitemap feed by lastmod time.	2020-11-06 09:30:15 +13:00
Jason	ee49d2021e	newsroom	2020-11-05 20:28:55 +00:00
Jason	c391c50ab1	use localize	2020-11-05 04:15:31 +00:00
Jason Schwarzenberger	095f0d549a	use replace.	2020-11-05 16:57:08 +13:00
Jason Schwarzenberger	c21c71667e	fix date issue.	2020-11-05 16:41:15 +13:00
Jason Schwarzenberger	c3a2c91a11	update requirements.txt	2020-11-05 16:33:50 +13:00
Jason Schwarzenberger	0f39446a61	tz aware for use in settings.	2020-11-05 16:30:55 +13:00
Jason Schwarzenberger	351059aab1	fix excludes.	2020-11-05 15:59:13 +13:00
Jason Schwarzenberger	4488e2c292	add an excludes list of substrings for urls in the settings for sitemap/category.	2020-11-05 15:51:59 +13:00
Jason Schwarzenberger	afda5b635c	disqus test.	2020-11-05 14:23:51 +13:00
Jason Schwarzenberger	0fc1a44d2b	fix issue in substack.	2020-11-04 17:40:29 +13:00
Jason Schwarzenberger	9fff1b9e46	avoid duplicate articles listed on the category page	2020-11-04 17:14:42 +13:00
Jason Schwarzenberger	16b59f6c67	try stop bad pages.	2020-11-04 16:34:31 +13:00
Jason Schwarzenberger	939f4775a7	better settings example.	2020-11-04 15:52:34 +13:00
Jason Schwarzenberger	9bfc6fc6fa	scraper settings, ordering and loop.	2020-11-04 15:47:12 +13:00
Jason Schwarzenberger	6ea9844d00	remove useless try blocks.	2020-11-04 15:37:19 +13:00
Jason Schwarzenberger	1318259d3d	imply referrer is substack.	2020-11-04 15:21:07 +13:00
Jason Schwarzenberger	98a0c2257c	increase declutter timeout.	2020-11-04 15:15:00 +13:00
Jason Schwarzenberger	e6976db25d	fix tabs	2020-11-04 15:04:20 +13:00
Jason Schwarzenberger	9edc8b7cca	move scraping for article content to files.	2020-11-04 15:00:58 +13:00
Jason Schwarzenberger	33e21e7f30	fix mistake.	2020-11-04 12:45:01 +13:00
Jason Schwarzenberger	892a99eca6	add + expander in place of collapser.	2020-11-04 12:43:15 +13:00
Jason Schwarzenberger	d718d05a04	fix dates for newsroom.	2020-11-04 11:53:16 +13:00
Jason Schwarzenberger	d1795eb1b8	add radionz and newsroom logos.	2020-11-04 11:30:56 +13:00
Jason Schwarzenberger	9f4ff4acf0	remove unnecessary sitemap.xml request.	2020-11-04 11:22:15 +13:00
Jason Schwarzenberger	db6aad84ec	fix mistake.	2020-11-04 11:12:01 +13:00