Compare commits

..

243 Commits

Author SHA1 Message Date
tanner 7c600dcfba Only wrap code in comments 2025-12-03 04:18:36 +00:00
tanner 92e70229fe fix: Refine code block detection to ignore inline <code>
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 03:57:08 +00:00
tanner b749e58f62 fix: Refine code block detection to exclude inline code
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 03:55:18 +00:00
tanner b1b2be6080 fix: Use textContent for code block conversion to prevent content loss
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 03:51:33 +00:00
tanner 5ebe87dbc2 refactor: Optimize nodes() calls and simplify function in Article
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 03:50:10 +00:00
tanner a8a36b693e fix: Render void elements correctly and copy all attributes
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 03:12:51 +00:00
tanner 60eefb4b27 refactor: Implement recursive rendering to detect and convert code blocks
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 02:52:07 +00:00
tanner 8f5dae4bdc fix: Unwrap single-child wrapper elements in nodes function
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 02:46:20 +00:00
tanner 89a511efc0 chore: Add debug log to isCodeBlock function 2025-12-03 02:46:18 +00:00
tanner 504fe745ea fix: Relax isCodeBlock check for nested code elements
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 02:37:58 +00:00
tanner 762e8a9a2e refactor: Refactor nodes logic from useMemo to a regular function 2025-12-03 02:37:56 +00:00
tanner 6dc47f6672 refactor: Extract code block detection into isCodeBlock function
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 01:46:19 +00:00
tanner da108f25d4 fix: Detect code blocks nested in pre tags for conversion
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 01:43:33 +00:00
tanner a2303841ec fix: Show 'Convert Code to Paragraph' button for <code> elements
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 01:37:08 +00:00
tanner 0e7aedbc5e fix: Adjust spacing below comment text content
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 01:28:10 +00:00
tanner ec7d395407 fix: Wrap text in <pre> blocks to prevent horizontal overflow
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 00:58:39 +00:00
tanner fd5acd4861 refactor: Convert 'show more' div to semantic button
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 00:50:58 +00:00
tanner b1d4fc2903 refactor: Convert collapser span to button for accessibility
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-03 00:48:22 +00:00
tanner 0f87d47536 refactor: Remove unnecessary useCallback from comment functions
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 23:53:40 +00:00
tanner 8472907730 Mark deleted / empty comments 2025-12-02 23:39:24 +00:00
tanner 482753e96a Add a copy button to the article title 2025-12-02 23:19:31 +00:00
tanner 169a84faa1 fix: Align article title and copy button, correct icon font
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 23:19:31 +00:00
tanner 6fa929fb1f style: Update copy link button font 2025-12-02 23:19:31 +00:00
tanner 5f02a95cf3 fix: Improve copy button icon display and alignment
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 23:19:31 +00:00
tanner 1789f88d4d style: Style copy button icon 2025-12-02 23:19:31 +00:00
tanner f5eab47496 feat: Use icons for copy link button feedback
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 23:19:31 +00:00
tanner 985e596790 feat: Add button to copy article title and URL to clipboard
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 23:19:31 +00:00
tanner 30298928f3 Move static build directory to apiserver/ 2025-12-02 22:38:49 +00:00
tanner 8d7d692d9c refactor: Iterate through stories in order for prioritized updates
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 22:37:58 +00:00
tanner bd85127613 fix: Unregister service worker
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 17:13:52 +00:00
tanner 4c9d5eede1 Revert ScrollToTop component back to class-based 2025-12-02 17:02:03 +00:00
tanner bf3e6bbc28 Don't setStories every loop iteration 2025-12-02 16:52:32 +00:00
tanner 856c360d98 feat: Add loading progress indicator to Feed
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-12-02 01:20:27 +00:00
tanner 1ce55e6d1f feat: Add fetching stories placeholder 2025-12-02 01:20:25 +00:00
tanner 6a329e3ba9 Misc fixes 2025-12-01 21:07:01 +00:00
tanner 3acaf230c4 fix: Improve submit error handling on API and refactor client with async/await
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 23:02:29 +00:00
tanner 7b84573dd8 fix: Improve error handling for non-JSON server responses in Submit
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 22:59:15 +00:00
tanner 7523426f15 feat: Display detailed submission errors to user
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 22:56:48 +00:00
tanner b2ec85cfa5 feat: Display detailed, expandable connection error in Comments component
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 22:51:14 +00:00
tanner 8c201d5c2e fix: Conditionally render error details to avoid layout gap
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 22:45:58 +00:00
tanner a21c84efc6 refactor: Improve article loading error and cache messages 2025-11-21 22:45:54 +00:00
tanner 15aa413584 fix: Prevent layout shift when error message appears
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 22:39:34 +00:00
tanner e9ee231954 feat: Persist new stories and improve layout consistency 2025-11-21 22:39:32 +00:00
tanner 62d5915133 feat: Add detailed, expandable error messages to Article component
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 22:34:24 +00:00
tanner 61ec583882 feat: Show preload progress on fetch failure
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 00:59:14 +00:00
tanner 1443fdcc32 style: Improve error messages and loading text, add spacing to error details 2025-11-21 00:59:12 +00:00
tanner f2310b6925 fix: Provide detailed error for story fetch failures
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 00:50:58 +00:00
tanner aa80570da4 fix: Display network error on API fetch failure
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 00:49:14 +00:00
tanner 7d0e60f5f0 fix: Provide detailed error messages for network failures
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 00:45:59 +00:00
tanner 21b5d67052 feat: Show detailed connection errors in collapsible section 2025-11-21 00:41:57 +00:00
tanner 53468c8ccd feat: Add 10s timeout and early exit for story preloading on error
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-21 00:34:17 +00:00
tanner 6cfb4b317f feat: Immediately display stories on first load
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-20 23:02:59 +00:00
tanner f08202d592 fix: Always fetch full story and update existing in feed
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-20 22:58:44 +00:00
tanner 5a7f55184d Begin stats API route 2025-11-20 22:25:26 +00:00
tanner e84062394b Ignore aider files 2025-11-20 22:25:20 +00:00
tanner e867d5d868 Add debug logging, debug add manual submissions to feed 2025-11-20 21:55:45 +00:00
tanner 845d87ec55 Logging 2025-11-19 19:17:38 +00:00
tanner e18aaad741 fix: Batch story list updates and limit length
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-19 19:17:38 +00:00
tanner 02e86efb4f chore: Add console log for stories 2025-11-19 19:17:38 +00:00
tanner b85d879ae7 fix: Fix infinite loop in Feed by removing stories from useEffect deps
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-19 19:17:38 +00:00
tanner 55bf75742e refactor: Refactor Feed story fetching for improved network resilience
Co-authored-by: aider (gemini/gemini-2.5-pro) <aider@aider.chat>
2025-11-19 19:17:38 +00:00
tanner 83cb6fc0ae chore: Disable story updates and preloading logic 2025-11-19 19:17:38 +00:00
tanner 667c2c5eaf refactor: Refactor dot components to functional 2025-11-19 19:17:38 +00:00
tanner 1df1c59d61 refactor: Refactor Submit component to use hooks 2025-11-19 19:17:38 +00:00
tanner c4f2e7d595 refactor: Refactor Search component to use hooks 2025-11-19 19:17:38 +00:00
tanner f61cfc09b0 refactor: Convert ScrollToTop to functional component with hooks 2025-11-19 19:17:38 +00:00
tanner 366e76e25d refactor: refactor Results component to functional component 2025-11-19 19:17:38 +00:00
tanner 6f1811c564 Update webclient dependencies 2025-11-19 19:17:38 +00:00
tanner 443115ac0f refactor: Refactor Feed component to functional with hooks 2025-11-19 19:17:38 +00:00
tanner 034c440e46 refactor: Convert Comments class to functional using hooks 2025-11-19 19:17:38 +00:00
tanner 26a6353ca5 refactor: Rename Article component to Comments 2025-11-19 19:17:38 +00:00
tanner 7ac4dfa01c refactor: Refactor Article component to use hooks 2025-11-19 19:17:38 +00:00
tanner 633429c976 refactor: Convert App class component to functional component 2025-11-19 19:17:38 +00:00
tanner 5cdbf6ef54 Ignore blank hackernews titles 2025-11-19 19:17:38 +00:00
tanner f1a30d0af2 Skip "Removed by moderator" stories 2025-09-27 17:38:50 +00:00
tanner 9ec61ea5bc Ignore dead and political stories 2025-05-27 18:47:17 +00:00
tanner bdc7a6c10d Fix Better HN api content extraction 2025-02-01 22:39:13 +00:00
tanner 4858516b01 Add Better HN as an API backup 2025-02-01 21:42:06 +00:00
tanner f10e6063fc Bug fixes 2025-02-01 20:31:35 +00:00
tanner 249a616531 Alert on story update error 2024-03-16 20:41:24 +00:00
tanner ab92bd5441 Adjust score and comment thresholds 2024-03-08 03:08:18 +00:00
tanner 6b16a768a7 Fix deletion script 2024-03-08 03:08:03 +00:00
tanner 57de076fec Increase database timeout 2024-02-27 18:48:56 +00:00
tanner 074b898508 Fix lobsters comment parsing 2024-02-27 18:47:00 +00:00
tanner f049d194ab Move scripts into own folder 2024-02-27 18:32:29 +00:00
tanner c2b9a1cb7a Update readability 2024-02-27 18:32:19 +00:00
tanner 4435f49e17 Make "dark" theme grey, add "black" theme 2023-09-13 01:19:47 +00:00
tanner 494d89ac30 Disable lobsters 2023-09-13 01:02:15 +00:00
tanner e79fca6ecc Replace "indent_level" with "depth" in lobsters API
See:
https://github.com/lobsters/lobsters/commit/fe09e5aa31993e09ed4ad255bb4a359f1e8a2d62
2023-08-31 07:35:44 +00:00
tanner c65fb69092 Handle Lobsters comment parsing TypeErrors
Too lazy to debug this:

2023-08-29 12:56:35,111 - root - INFO - Updating lobsters story: yktkwr, index: 55
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 854, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/tanner/qotnews/apiserver/server.py", line 194, in feed_thread
    valid = feed.update_story(story)
  File "/home/tanner/qotnews/apiserver/feed.py", line 74, in update_story
    res = lobsters.story(story['ref'])
  File "/home/tanner/qotnews/apiserver/feeds/lobsters.py", line 103, in story
    s['comments'] = iter_comments(r['comments'])
  File "/home/tanner/qotnews/apiserver/feeds/lobsters.py", line 76, in iter_comments
    parent_stack = parent_stack[:indent-1]
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
2023-08-29T12:56:35Z <Greenlet at 0x7f92ad840ae0: feed_thread> failed with TypeError
2023-08-31 07:30:39 +00:00
tanner 632d028e4c Add Tildes group whitelist 2023-07-13 22:54:36 +00:00
tanner ea8e9e5a23 Increase again 2023-06-13 17:11:50 +00:00
tanner 2838ea9b41 Increase Tildes story score requirement 2023-06-11 01:01:31 +00:00
tanner f15d108971 Catch all possible Reddit API exceptions 2023-03-15 21:16:37 +00:00
tanner f777348af8 Fix darkmode fullscreen button color 2022-08-11 19:36:36 +00:00
tanner 486404a413 Fix fix-stories bug 2022-08-10 04:06:39 +00:00
tanner 7c9c07a4cf Hide fullscreen button if it's not available 2022-08-10 04:05:25 +00:00
tanner 08d02f6013 Add fullscreen mode 2022-08-08 23:21:49 +00:00
tanner 1b54342702 Add red theme 2022-08-08 20:14:57 +00:00
tanner 9e9571a3c0 Write fixed stories to database 2022-07-05 00:57:56 +00:00
tanner dc83a70887 Begin script to fix bad gzip text 2022-07-04 20:32:01 +00:00
tanner 2e2c9ae837 Move FEED_LENGTH to settings.py, use for search results 2022-07-04 19:08:24 +00:00
tanner 61021d8f91 Small UI changes 2022-07-04 19:08:24 +00:00
tanner e65047fead Add accept gzip header to readability server 2022-07-04 19:07:31 +00:00
tanner 8e775c189f Add test file 2022-07-04 05:56:06 +00:00
tanner 3d9274309a Fix requests text encoding slowness 2022-07-04 05:55:52 +00:00
tanner 7bdbbf10b2 Return search results directly from the server 2022-07-04 04:33:01 +00:00
tanner 6aa0f78536 Remove Article / Comments, etc thing after name 2022-07-04 04:33:01 +00:00
tanner bf3663bbec Remove hard-coded title 2022-06-30 00:12:22 +00:00
tanner e6589dc61c Adjust title 2022-06-30 00:05:15 +00:00
tanner 307e8349f3 Change header based on page 2022-06-30 00:00:30 +00:00
tanner 04cd56daa8 Add index / noindex to client 2022-06-29 23:30:39 +00:00
tanner c80769def6 Add noindex meta tag to stories 2022-06-29 23:20:53 +00:00
tanner ebd1ad2140 Increase database timeout 2022-06-24 20:50:27 +00:00
tanner 2cc7dd0d6d Update software 2022-05-31 04:24:12 +00:00
tanner 6e7cb86d2e Explain no javascript 2022-05-31 04:23:52 +00:00
tanner a25457254f Improve logging, sends tweets to nitter.net 2022-03-05 23:48:46 +00:00
tanner a693ea5342 Remove outline API 2022-03-05 22:05:29 +00:00
tanner 7386e1d8b0 Include option to disable readerserver 2022-03-05 22:04:25 +00:00
tanner f8e8597e3a Include option to disable search 2022-03-05 21:58:35 +00:00
tanner 55c282ee69 Fix search to work with low-RAM server 2022-03-05 21:33:07 +00:00
tanner 3f774a9e38 Improve logging 2021-09-06 00:21:05 +00:00
tanner dcedd4caa1 Add script to reindex search, abstract search API 2021-09-06 00:20:21 +00:00
tanner 7a131ebd03 Change the order by which content-type is grabbed 2021-01-30 06:36:02 +00:00
tanner 6f64401785 Add optional skip and limit to API route 2021-01-18 03:59:33 +00:00
tanner 3ff917e806 Remove colons from date string so Python 3.5 can parse 2020-12-15 23:19:50 +00:00
tanner c9fb9bd5df Add Lobsters to feed 2020-12-12 05:26:33 +00:00
tanner fd9c9c888d Update gitignore 2020-12-11 23:49:45 +00:00
tanner 42dcf15374 Increase sqlite lock timeout 2020-11-19 21:38:18 +00:00
tanner d8a0b77765 Blacklist sec.gov website 2020-11-19 21:37:59 +00:00
tanner 9a279d44b1 Add header to get content type 2020-11-03 20:27:43 +00:00
tanner e506804666 Clean code up 2020-11-03 03:45:56 +00:00
tanner ca78a6d7a9 Move feed and Praw config to settings.py 2020-11-02 02:26:54 +00:00
tanner 7acce407e9 Fix index.html indentation 2020-11-02 00:38:34 +00:00
tanner 5281672000 Fix noscript font color 2020-11-02 00:36:11 +00:00
tanner e59acefda9 Remove Whoosh 2020-11-02 00:22:40 +00:00
tanner cbc802b7e9 Try Hackernews API twice 2020-11-02 00:17:22 +00:00
tanner 4579dfce00 Improve logging 2020-11-02 00:13:43 +00:00
tanner 0d16bec6f6 Fix table width CSS 2020-11-01 00:47:18 +00:00
tanner feba8b7aa0 Make qotnews work with WaPo 2020-10-29 04:55:34 +00:00
tanner ee5105743d Upgrade readability 2020-10-29 01:24:13 +00:00
tanner 72802a6fcf Show exerpt of hidden comments 2020-10-27 00:41:36 +00:00
tanner 99d3a234f4 Fix bug with rendering text nodes 2020-10-26 21:58:36 +00:00
tanner f95df227f1 Add instructions to download search server 2020-10-26 21:58:36 +00:00
tanner b82095ca7a Add buttons to collapse / expand comments 2020-10-26 21:57:10 +00:00
tanner 992c1c1233 Monkeypatch earlier 2020-10-24 22:30:00 +00:00
tanner 88d2216627 Add a script to delete a story 2020-10-03 23:42:21 +00:00
tanner 6cf2f01b08 Adjust feeds 2020-10-03 23:41:57 +00:00
tanner 607573dd44 Add buttons to convert <pre> to <p> 2020-10-03 23:23:25 +00:00
tanner c554ecd890 Add a line on UI to make search results obvious 2020-08-14 03:58:11 +00:00
tanner 6576eb1bac Adjust content-type request timeout 2020-08-14 03:57:43 +00:00
tanner 472af76d1a Adjust port 2020-08-14 03:57:18 +00:00
tanner 4727d34eb6 Delete displayed-attributes when init search 2020-08-14 03:56:47 +00:00
tanner 0e086b60b8 Remove business subreddit from feed 2020-08-14 03:55:28 +00:00
tanner b46ce36c63 Update requirements 2020-07-08 05:24:32 +00:00
tanner 9a449bf3ca Remove extra logging 2020-07-08 02:36:40 +00:00
tanner 0bd9f05250 Fix crash when HN feed fails 2020-07-08 02:36:40 +00:00
tanner 9c116bde4a Remove document img and ignore r/technology 2020-07-08 02:36:40 +00:00
tanner ebedaef00b Tune search rankings and attributes 2020-07-08 02:36:40 +00:00
tanner d7f0643bd7 Add more logging 2020-07-08 02:36:40 +00:00
tanner eb1137299d Remove article numbers 2020-07-08 02:36:40 +00:00
tanner 72d4a68929 Remove pre-fetching image 2020-07-08 02:36:40 +00:00
tanner f1c846acd0 Remove get first image 2020-07-08 02:36:40 +00:00
tanner 850b30e353 Add requests timeouts and temporary logging 2020-07-08 02:36:40 +00:00
tanner d614ad0743 Integrate with external MeiliSearch server 2020-07-08 02:36:40 +00:00
tanner f46cafdc90 Integrate sqlite database with server 2020-07-08 02:36:40 +00:00
tanner 873dc44cb1 Update whoosh migration script 2020-07-08 02:36:40 +00:00
tanner 1fb9db3f4b Store ref list in database too 2020-07-08 02:36:40 +00:00
tanner b923908a45 Begin initial sqlite conversion 2020-07-08 02:36:40 +00:00
tanner dbdcfaa921 Check if cache is broken 2020-07-08 02:36:40 +00:00
tanner 8799b10525 Fall back to ref on manual submission title 2020-07-08 02:36:40 +00:00
tanner 6430fe5e9f Check content-type 2020-07-08 02:36:40 +00:00
tanner a4cf719cb8 Remove technology subreddit 2020-07-08 02:36:40 +00:00
tanner 595f469b4a Update tildes parser group tag 2020-07-08 02:36:40 +00:00
tanner b252c6a207 Make noscript background white 2020-06-22 20:52:51 +00:00
tanner 02b73a8b14 Fix cache load race condition bug 2020-01-28 04:20:48 +00:00
tanner 72f1043952 Remove preload of news source icons 2020-01-28 04:20:29 +00:00
tanner 7b31fcf690 Remove keys of uncached stories 2020-01-28 04:20:05 +00:00
tanner b3d2eeb67f Fix tildes deleted comment parser error 2020-01-28 04:19:26 +00:00
tanner 9078b567f0 Add del tag and sort tags 2020-01-04 23:37:41 +00:00
tanner ced20390eb Fix back/forward scroll jump issue 2020-01-04 23:36:24 +00:00
tanner 6cd41f0902 Add forward button, convert icons to font 2020-01-03 03:45:56 +00:00
tanner 746932ab96 Add style changes to prevent horizontal scrolling 2019-12-22 21:43:33 +00:00
tanner 2822974b6e Stop using archive.is on articles (hits CAPTCHAs) 2019-12-15 22:47:33 +00:00
tanner 8fd7fc158c Fix search result icons 2019-12-14 07:39:25 +00:00
tanner 17ef7e3a65 Whitelist more html tags 2019-12-14 07:39:10 +00:00
tanner 3363ccd47e Embed base64 logo directly in source to avoid load 2019-12-02 23:54:02 +00:00
tanner 2d80b19414 Grab comments on manually submitted links 2019-12-02 23:15:51 +00:00
tanner ebcbf1b624 Sanitize html 2019-12-01 22:18:41 +00:00
tanner e231cd5c31 Decrease feed cache length to 150 2019-12-01 22:18:14 +00:00
tanner 569e5b16ca Add logo for manual submissions 2019-11-14 08:38:11 +00:00
tanner db5097ac57 Drop articles more than two days old 2019-11-08 21:50:33 +00:00
tanner 2edb3ceba7 Allow manual submission of articles 2019-11-08 05:55:30 +00:00
tanner 38b5f2dbeb Move to gevent production http server 2019-11-08 02:37:57 +00:00
tanner 6826f731c7 Handle hostnames better 2019-11-07 22:10:08 +00:00
tanner bb693ba434 Add subreddit 2019-11-07 22:09:45 +00:00
tanner 632b0276c4 Abort previous search requests 2019-11-07 22:08:28 +00:00
tanner 4cf97304e4 Get rid of lint warnings 2019-10-22 07:31:59 +00:00
tanner 9e55f6e4ec Fix Tildes down for maintenance edge case 2019-10-22 05:01:30 +00:00
tanner edc4c439d7 Prefetch first images 2019-10-19 07:33:06 +00:00
tanner 187c6b8110 Cache articles in memory for speed 2019-10-18 21:26:22 +00:00
tanner 6764bf0d6d Add serviceworker, render logos directly 2019-10-18 05:09:49 +00:00
tanner dc588fee91 Fix underlines 2019-10-18 01:20:38 +00:00
tanner f8998b687e Fix crash from domain and ext check bug 2019-10-16 08:56:31 +00:00
tanner e4f81472fc Fix copy/paste error, switch to info logging 2019-10-16 05:26:47 +00:00
tanner f293f2b5f9 Begin README and add license 2019-10-15 16:40:55 -06:00
tanner 810e8c5ead Archive WSJ articles first, catch KeyboardInterrupt 2019-10-15 21:03:47 +00:00
tanner 9c4766a928 Stop using python keyword id for id 2019-10-15 20:36:20 +00:00
tanner 0f5b2a5ff9 Cache all articles in IndexedDB 2019-10-12 23:41:31 +00:00
tanner 7cb87b59fe Move archive to Whoosh and add search 2019-10-12 05:32:17 +00:00
tanner 45b75b420b Gitkeep archive directory 2019-10-10 21:55:21 +00:00
tanner f0721519e1 Serve client through apiserver, adding meta info 2019-10-10 21:54:29 +00:00
tanner 25a671f58e Set title on article and comment pages, add comment anchors 2019-10-10 21:52:28 +00:00
tanner 5fd4fdb21c Fix Tildes comments with unknown authors 2019-10-08 08:01:17 +00:00
tanner 19e9a80be1 Archive Bloomberg articles first 2019-10-08 08:00:50 +00:00
tanner 5caa4542d8 Gitkeep apiserver data directory 2019-10-08 07:59:30 +00:00
tanner 1ed2baded6 Add huge margin to bottom of body for better pagescroll 2019-09-24 18:40:22 +00:00
tanner c7734eb2bc Add site logos, keep displaying news on error 2019-09-24 08:23:14 +00:00
tanner 0053147226 Ignore certain files and domains, remove refs 2019-09-24 08:22:06 +00:00
tanner 0496fbba45 Ignore new Tildes posts and handle deleted ones 2019-09-24 08:21:26 +00:00
tanner 0a1ebaa8b8 Handle Reddit PRAW exceptions 2019-09-24 08:20:46 +00:00
tanner 2ede5ed6ff Filter out False comments 2019-08-30 06:23:14 +00:00
tanner 20a9d9d452 Settle on serif font, add scroll to top component 2019-08-30 06:22:26 +00:00
tanner 23cdbc9292 Render reddit markdown, poll tildes better, add utils 2019-08-28 04:13:02 +00:00
tanner 10d4ec863e Snip deeply nested comments 2019-08-26 01:37:50 +00:00
tanner fc8ce79e33 Try outline.com for reader mode first 2019-08-25 23:49:08 +00:00
tanner 8eca354a47 Add favicons to webclient 2019-08-25 23:48:24 +00:00
tanner b1275d9a27 Add a button to toggle between article and comments 2019-08-25 08:50:49 +00:00
tanner 9336760ed3 Add fonts, fix styling issues 2019-08-25 07:46:58 +00:00
tanner cf9e197e6c Fix tildes comments parsing bug 2019-08-25 07:46:22 +00:00
tanner 2b1a352917 Clear localstorage cache and add slogan 2019-08-25 01:25:28 +00:00
tanner 1b6c8fc6cb Add tildes to feeds 2019-08-25 00:36:26 +00:00
tanner a2509958da Add reddit to feeds 2019-08-24 21:37:43 +00:00
tanner 4450e93c65 Remove DOMPurify import 2019-08-24 08:49:53 +00:00
tanner d341d4422f Abstract api server feeds 2019-08-24 08:49:11 +00:00
tanner 82074eb8aa Stop running DOMPurify on reader server 2019-08-24 05:09:02 +00:00
tanner c1a81a4d8c Write news stories to disk 2019-08-24 05:07:16 +00:00
tanner dde6ac4566 Finish prototype web client 2019-08-24 05:04:51 +00:00
tanner 62d68da415 Finish prototype api server 2019-08-23 08:23:48 +00:00
tanner c04b5c27f2 Figure out .gitignores 2019-08-23 08:23:26 +00:00
tanner 771c3987ec Change reader server useragent and port 2019-08-23 08:21:25 +00:00
tanner c0607b3fb6 Prototype readability server 2019-08-20 21:49:06 -06:00
tanner a814411c12 Initial commit 2019-08-20 21:48:55 -06:00
26 changed files with 68 additions and 478 deletions
+1 -1
View File
@@ -1,6 +1,6 @@
The MIT License (MIT) The MIT License (MIT)
Copyright (c) 2019 Tanner (tanner.vc) Copyright (c) 2019 Tanner Collin
Permission is hereby granted, free of charge, to any person obtaining a copy Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal of this software and associated documentation files (the "Software"), to deal
+1 -1
View File
@@ -20,7 +20,7 @@ $ sudo apt install yarn
Clone this repo: Clone this repo:
```text ```text
$ git clone https://git.tanner.vc/tanner/qotnews.git $ git clone https://gogs.tannercollin.com/tanner/qotnews.git
$ cd qotnews $ cd qotnews
``` ```
+1 -1
View File
@@ -16,7 +16,7 @@ from utils import clean
# cache the topic groups to prevent redirects # cache the topic groups to prevent redirects
group_lookup = {} group_lookup = {}
USER_AGENT = 'qotnews scraper (github:tanner37)' USER_AGENT = 'qotnews scraper (github:tannercollin)'
API_TOPSTORIES = lambda : 'https://tildes.net' API_TOPSTORIES = lambda : 'https://tildes.net'
API_ITEM = lambda x : 'https://tildes.net/shortener/{}'.format(x) API_ITEM = lambda x : 'https://tildes.net/shortener/{}'.format(x)
-67
View File
@@ -1,67 +0,0 @@
import logging
logging.basicConfig(
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
level=logging.INFO)
import database
from sqlalchemy import select
import search
import sys
import time
import json
import requests
from bs4 import BeautifulSoup
database.init()
search.init()
BATCH_SIZE = 1000
def put_stories(stories):
return search.meili_api(requests.post, 'indexes/qotnews/documents', stories)
def get_update(update_id):
return search.meili_api(requests.get, 'tasks/{}'.format(update_id))
if __name__ == '__main__':
num_stories = database.count_stories()
print('Reindex {} stories?'.format(num_stories))
print('Press ENTER to continue, ctrl-c to cancel')
input()
story_list = database.get_story_list()
count = 1
while len(story_list):
stories = []
for _ in range(BATCH_SIZE):
try:
sid = story_list.pop()
except IndexError:
break
story = database.get_story(sid)
print('Indexing {}/{} id: {} title: {}'.format(count, num_stories, sid[0], story.title))
story_obj = json.loads(story.full_json)
story_obj.pop('comments', False)
if 'text' in story_obj and story_obj['text']:
soup = BeautifulSoup(story_obj['text'], 'html.parser')
story_obj['text'] = soup.get_text()
stories.append(story_obj)
count += 1
res = put_stories(stories)
update_id = res['taskUid']
print('Waiting for processing', end='')
while get_update(update_id)['status'] != 'succeeded':
time.sleep(0.5)
print('.', end='', flush=True)
print()
print('Done.')
-1
View File
@@ -8,7 +8,6 @@ Flask==1.1.2
Flask-Cors==3.0.8 Flask-Cors==3.0.8
gevent==20.6.2 gevent==20.6.2
greenlet==0.4.16 greenlet==0.4.16
humanize==4.10.0
idna==2.10 idna==2.10
itsdangerous==1.1.0 itsdangerous==1.1.0
Jinja2==2.11.2 Jinja2==2.11.2
-4
View File
@@ -11,7 +11,6 @@ import sys
import time import time
import json import json
import requests import requests
from bs4 import BeautifulSoup
database.init() database.init()
search.init() search.init()
@@ -46,9 +45,6 @@ if __name__ == '__main__':
story = database.get_story(sid) story = database.get_story(sid)
print('Indexing {}/{} id: {} title: {}'.format(count, num_stories, sid[0], story.title)) print('Indexing {}/{} id: {} title: {}'.format(count, num_stories, sid[0], story.title))
story_obj = json.loads(story.meta_json) story_obj = json.loads(story.meta_json)
if 'text' in story_obj and story_obj['text']:
soup = BeautifulSoup(story_obj['text'], 'html.parser')
story_obj['text'] = soup.get_text()
stories.append(story_obj) stories.append(story_obj)
count += 1 count += 1
+23 -22
View File
@@ -10,8 +10,7 @@ SEARCH_ENABLED = bool(settings.MEILI_URL)
def meili_api(method, route, json=None, params=None, parse_json=True): def meili_api(method, route, json=None, params=None, parse_json=True):
try: try:
headers = {'Authorization': 'Bearer ' + settings.MEILI_API_KEY} r = method(settings.MEILI_URL + route, json=json, params=params, timeout=4)
r = method(settings.MEILI_URL + route, json=json, params=params, headers=headers, timeout=4)
if r.status_code > 299: if r.status_code > 299:
raise Exception('Bad response code ' + str(r.status_code)) raise Exception('Bad response code ' + str(r.status_code))
if parse_json: if parse_json:
@@ -25,40 +24,42 @@ def meili_api(method, route, json=None, params=None, parse_json=True):
logging.error('Problem with MeiliSearch api route: %s: %s', route, str(e)) logging.error('Problem with MeiliSearch api route: %s: %s', route, str(e))
return False return False
def update_settings(): def create_index():
json = { json = dict(uid='qotnews', primaryKey='id')
'rankingRules': ['words', 'typo', 'proximity', 'attribute', 'date:desc', 'exactness'], return meili_api(requests.post, 'indexes', json=json)
'searchableAttributes': ['title', 'url', 'author', 'text'],
'displayedAttributes': ['id', 'ref', 'source', 'author', 'author_link', 'score', 'date', 'title', 'link', 'url', 'num_comments', 'text'], def update_rankings():
'stopWords': ['a', 'an', 'the', 'and', 'or', 'but', 'if', 'in', 'on', 'at', 'by', 'for', 'with', 'to', 'from', 'of', 'is', 'it', 'that', 'this'], json = ['typo', 'words', 'proximity', 'date:desc', 'exactness']
} return meili_api(requests.post, 'indexes/qotnews/settings/ranking-rules', json=json)
return meili_api(requests.patch, 'indexes/qotnews/settings', json=json)
def update_attributes():
json = ['title', 'url', 'author']
r = meili_api(requests.post, 'indexes/qotnews/settings/searchable-attributes', json=json)
json = ['id', 'ref', 'source', 'author', 'author_link', 'score', 'date', 'title', 'link', 'url', 'num_comments']
r = meili_api(requests.post, 'indexes/qotnews/settings/displayed-attributes', json=json)
return r
def init(): def init():
if not SEARCH_ENABLED: if not SEARCH_ENABLED:
logging.info('Search is not enabled, skipping init.') logging.info('Search is not enabled, skipping init.')
return return
update_settings() print(create_index())
update_rankings()
update_attributes()
def put_story(story): def put_story(story):
if not SEARCH_ENABLED: return if not SEARCH_ENABLED: return
return meili_api(requests.post, 'indexes/qotnews/documents', [story]) return meili_api(requests.post, 'indexes/qotnews/documents', [story])
def search(q, in_article=False): def search(q):
if not SEARCH_ENABLED: return [] if not SEARCH_ENABLED: return []
params = dict(q=q, limit=settings.FEED_LENGTH)
json = dict(q=q, limit=settings.FEED_LENGTH) r = meili_api(requests.get, 'indexes/qotnews/search', params=params, parse_json=False)
if True:
json['attributesToSearchOn'] = ['text']
json['attributesToCrop'] = ['text']
json['attributesToRetrieve'] = ['id', 'ref', 'source', 'author', 'author_link', 'score', 'date', 'title', 'link', 'url', 'num_comments']
json['cropLength'] = 80
r = meili_api(requests.post, 'indexes/qotnews/search', json=json, parse_json=False)
return r return r
if __name__ == '__main__': if __name__ == '__main__':
init() init()
print(update_rankings())
print(search('facebook')) print(search('facebook'))
+7 -78
View File
@@ -14,9 +14,6 @@ import json
import threading import threading
import traceback import traceback
import time import time
import datetime
import humanize
import urllib.request
from urllib.parse import urlparse, parse_qs from urllib.parse import urlparse, parse_qs
import settings import settings
@@ -29,25 +26,6 @@ from flask import abort, Flask, request, render_template, stream_with_context, R
from werkzeug.exceptions import NotFound from werkzeug.exceptions import NotFound
from flask_cors import CORS from flask_cors import CORS
smallweb_set = set()
def load_smallweb_list():
EXCLUDED = [
'github.com',
]
global smallweb_set
try:
url = 'https://raw.githubusercontent.com/kagisearch/smallweb/refs/heads/main/smallweb.txt'
with urllib.request.urlopen(url, timeout=10) as response:
urls = response.read().decode('utf-8').splitlines()
hosts = {urlparse(u).hostname for u in urls if u and urlparse(u).hostname}
smallweb_set = {h.replace('www.', '') for h in hosts if h not in EXCLUDED}
logging.info('Loaded {} smallweb domains.'.format(len(smallweb_set)))
except Exception as e:
logging.error('Failed to load smallweb list: {}'.format(e))
load_smallweb_list()
database.init() database.init()
search.init() search.init()
@@ -61,54 +39,15 @@ def new_id():
nid = gen_rand_id() nid = gen_rand_id()
return nid return nid
def fromnow(ts):
return humanize.naturaltime(datetime.datetime.fromtimestamp(ts))
build_folder = './build' build_folder = './build'
flask_app = Flask(__name__, template_folder=build_folder, static_folder=build_folder, static_url_path='') flask_app = Flask(__name__, template_folder=build_folder, static_folder=build_folder, static_url_path='')
flask_app.jinja_env.filters['fromnow'] = fromnow
cors = CORS(flask_app) cors = CORS(flask_app)
@flask_app.route('/api') @flask_app.route('/api')
def api(): def api():
skip = request.args.get('skip', 0) skip = request.args.get('skip', 0)
limit = request.args.get('limit', settings.FEED_LENGTH) limit = request.args.get('limit', settings.FEED_LENGTH)
stories = database.get_stories(limit, skip)
if request.args.get('smallweb') == 'true' and smallweb_set:
limit = int(limit)
skip = int(skip)
filtered_stories = []
current_skip = skip
while len(filtered_stories) < limit:
stories_batch = database.get_stories(limit, current_skip)
if not stories_batch:
break
for story_str in stories_batch:
story = json.loads(story_str)
story_url = story.get('url') or story.get('link') or ''
if not story_url:
continue
hostname = urlparse(story_url).hostname
if hostname:
hostname = hostname.replace('www.', '')
if hostname in smallweb_set:
filtered_stories.append(story_str)
if len(filtered_stories) == limit:
break
if len(filtered_stories) == limit:
break
current_skip += limit
stories = filtered_stories
else:
stories = database.get_stories(limit, skip)
# hacky nested json # hacky nested json
res = Response('{"stories":[' + ','.join(stories) + ']}') res = Response('{"stories":[' + ','.join(stories) + ']}')
res.headers['content-type'] = 'application/json' res.headers['content-type'] = 'application/json'
@@ -131,9 +70,8 @@ def apistats():
@flask_app.route('/api/search', strict_slashes=False) @flask_app.route('/api/search', strict_slashes=False)
def apisearch(): def apisearch():
q = request.args.get('q', '') q = request.args.get('q', '')
in_article = request.args.get('article', False)
if len(q) >= 3: if len(q) >= 3:
results = search.search(q, in_article) results = search.search(q)
else: else:
results = '[]' results = '[]'
res = Response(results) res = Response(results)
@@ -218,18 +156,11 @@ def story(sid):
@flask_app.route('/') @flask_app.route('/')
@flask_app.route('/search') @flask_app.route('/search')
def index(): def index():
stories_json = database.get_stories(settings.FEED_LENGTH, 0)
stories = [json.loads(s) for s in stories_json]
for s in stories:
url = urlparse(s.get('url') or s.get('link') or '').hostname or ''
s['hostname'] = url.replace('www.', '')
return render_template('index.html', return render_template('index.html',
title='QotNews', title='QotNews',
url='news.t0.vc', url='news.t0.vc',
description='Hacker News, Reddit, Lobsters, and Tildes articles rendered in reader mode', description='Hacker News, Reddit, Lobsters, and Tildes articles rendered in reader mode',
robots='index', robots='index',
stories=stories,
) )
@flask_app.route('/<sid>', strict_slashes=False) @flask_app.route('/<sid>', strict_slashes=False)
@@ -240,9 +171,9 @@ def static_story(sid):
except NotFound: except NotFound:
pass pass
story_obj = database.get_story(sid) story = database.get_story(sid)
if not story_obj: return abort(404) if not story: return abort(404)
story = json.loads(story_obj.full_json) story = json.loads(story.full_json)
score = story['score'] score = story['score']
num_comments = story['num_comments'] num_comments = story['num_comments']
@@ -251,7 +182,7 @@ def static_story(sid):
score, 's' if score != 1 else '', score, 's' if score != 1 else '',
num_comments, 's' if num_comments != 1 else '', num_comments, 's' if num_comments != 1 else '',
source) source)
url = urlparse(story.get('url') or story.get('link') or '').hostname or '' url = urlparse(story['url']).hostname or urlparse(story['link']).hostname or ''
url = url.replace('www.', '') url = url.replace('www.', '')
return render_template('index.html', return render_template('index.html',
@@ -259,11 +190,9 @@ def static_story(sid):
url=url, url=url,
description=description, description=description,
robots='noindex', robots='noindex',
story=story,
show_comments=request.path.endswith('/c'),
) )
http_server = WSGIServer(('0.0.0.0', 33842), flask_app) http_server = WSGIServer(('', 33842), flask_app)
def feed_thread(): def feed_thread():
global news_index, ref_list, current_item global news_index, ref_list, current_item
+1 -1
View File
@@ -12,7 +12,7 @@ def alert_tanner(message):
try: try:
logging.info('Alerting Tanner: ' + message) logging.info('Alerting Tanner: ' + message)
params = dict(qotnews=message) params = dict(qotnews=message)
requests.get('https://tbot.tanner.vc/message', params=params, timeout=4) requests.get('https://tbot.tannercollin.com/message', params=params, timeout=4)
except BaseException as e: except BaseException as e:
logging.error('Problem alerting Tanner: ' + str(e)) logging.error('Problem alerting Tanner: ' + str(e))
-1
View File
@@ -4,4 +4,3 @@
meilisearch-linux-amd64 meilisearch-linux-amd64
data.ms/ data.ms/
data.ms.old/

Before

Width:  |  Height:  |  Size: 538 B

After

Width:  |  Height:  |  Size: 538 B

Before

Width:  |  Height:  |  Size: 6.5 KiB

After

Width:  |  Height:  |  Size: 6.5 KiB

Before

Width:  |  Height:  |  Size: 5.4 KiB

After

Width:  |  Height:  |  Size: 5.4 KiB

Before

Width:  |  Height:  |  Size: 500 B

After

Width:  |  Height:  |  Size: 500 B

-2
View File
@@ -4,14 +4,12 @@
"private": true, "private": true,
"dependencies": { "dependencies": {
"abort-controller": "^3.0.0", "abort-controller": "^3.0.0",
"katex": "^0.16.25",
"localforage": "^1.7.3", "localforage": "^1.7.3",
"moment": "^2.24.0", "moment": "^2.24.0",
"query-string": "^6.8.3", "query-string": "^6.8.3",
"react": "^16.9.0", "react": "^16.9.0",
"react-dom": "^16.9.0", "react-dom": "^16.9.0",
"react-helmet": "^5.2.1", "react-helmet": "^5.2.1",
"react-latex-next": "^3.0.0",
"react-router-dom": "^5.0.1", "react-router-dom": "^5.0.1",
"react-router-hash-link": "^1.2.2", "react-router-hash-link": "^1.2.2",
"react-scripts": "3.1.1" "react-scripts": "3.1.1"
+17 -93
View File
@@ -35,105 +35,29 @@
overflow-y: scroll; overflow-y: scroll;
} }
body { body {
background: #eeeeee; background: #000;
}
.nojs {
color: white;
max-width: 32rem;
} }
</style> </style>
</head> </head>
<body> <body>
<div id="root"> <div class="nojs">
<div class="container menu"> <noscript>
<p> You need to enable JavaScript to run this app because it's written in React.
<a href="/">QotNews</a> I was planning on writing a server-side version, but I've become distracted
<br /> by other projects -- sorry!
<span class="slogan">Hacker News, Reddit, Lobsters, and Tildes articles rendered in reader mode.</span> <br/>
</p> I originally wrote this for myself, and of course I whitelist JavaScript on
</div> all my own domains.
{% if story %} <br/><br/>
<div class="{% if show_comments %}container{% else %}article-container{% endif %}"> Alternatively, try activex.news.t0.vc for an ActiveX™ version.
<div class="article"> </noscript>
<h1>{{ story.title }}</h1>
{% if show_comments %}
<div class="info">
<a href="/{{ story.id }}">View article</a>
</div>
{% else %}
<div class="info">
Source: <a class="source" href="{{ story.url or story.link }}">{{ url }}</a>
</div>
{% endif %}
<div class="info">
{{ story.score }} points
by <a href="{{ story.author_link }}">{{ story.author }}</a>
{{ story.date | fromnow }}
on <a href="{{ story.link }}">{{ story.source }}</a> |
<a href="/{{ story.id }}/c">
{{ story.num_comments }} comment{{ 's' if story.num_comments != 1 }}
</a>
</div>
{% if not show_comments and story.text %}
<div class="story-text">{{ story.text | safe }}</div>
{% elif show_comments %}
{% macro render_comment(comment, level) %}
<dt></dt>
<dd class="comment{% if level > 0 %} lined{% endif %}">
<div class="info">
<p>
{% if comment.author == story.author %}[OP] {% endif %}{{ comment.author or '[Deleted]' }} | <a href="#{{ comment.author }}{{ comment.date }}" id="{{ comment.author }}{{ comment.date }}">{{ comment.date | fromnow }}</a>
</p>
</div>
<div class="text">{{ (comment.text | safe) if comment.text else '<p>[Empty / deleted comment]</p>' }}</div>
{% if comment.comments %}
<dl>
{% for reply in comment.comments %}
{{ render_comment(reply, level + 1) }}
{% endfor %}
</dl>
{% endif %}
</dd>
{% endmacro %}
<dl class="comments">
{% for comment in story.comments %}{{ render_comment(comment, 0) }}{% endfor %}
</dl>
{% endif %}
</div>
<div class='dot toggleDot'>
<div class='button'>
<a href="/{{ story.id }}{{ '/c' if not show_comments else '' }}">
{{ '' if not show_comments else '' }}
</a>
</div>
</div>
</div>
{% elif stories %}
<div class="container">
{% for story in stories %}
<div class='item'>
<div class='title'>
<a class='link' href='/{{ story.id }}'>
<img class='source-logo' src='/logos/{{ story.source }}.png' alt='{{ story.source }}:' /> {{ story.title }}
</a>
<span class='source'>
(<a class='source' href='{{ story.url or story.link }}'>{{ story.hostname }}</a>)
</span>
</div>
<div class='info'>
{{ story.score }} points
by <a href="{{ story.author_link }}">{{ story.author }}</a>
{{ story.date | fromnow }}
on <a href="{{ story.link }}">{{ story.source }}</a> |
<a class="{{ 'hot' if story.num_comments > 99 else '' }}" href="/{{ story.id }}/c">
{{ story.num_comments }} comment{{ 's' if story.num_comments != 1 }}
</a>
</div>
</div>
{% endfor %}
</div>
{% endif %}
</div> </div>
<div id="root"></div>
<!-- <!--
This HTML file is a template. This HTML file is a template.
If you open it directly in the browser, you will see an empty page. If you open it directly in the browser, you will see an empty page.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 981 B

-38
View File
@@ -3,17 +3,8 @@ import { useParams } from 'react-router-dom';
import { Helmet } from 'react-helmet'; import { Helmet } from 'react-helmet';
import localForage from 'localforage'; import localForage from 'localforage';
import { sourceLink, infoLine, ToggleDot } from './utils.js'; import { sourceLink, infoLine, ToggleDot } from './utils.js';
import Latex from 'react-latex-next';
import 'katex/dist/katex.min.css';
const VOID_ELEMENTS = ['area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'link', 'meta', 'param', 'source', 'track', 'wbr']; const VOID_ELEMENTS = ['area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input', 'link', 'meta', 'param', 'source', 'track', 'wbr'];
const DANGEROUS_TAGS = ['svg', 'math'];
const latexDelimiters = [
{ left: '$$', right: '$$', display: true },
{ left: '\\[', right: '\\]', display: true },
{ left: '\\(', right: '\\)', display: false }
];
function Article({ cache }) { function Article({ cache }) {
const { id } = useParams(); const { id } = useParams();
@@ -104,11 +95,6 @@ function Article({ cache }) {
} }
if (v.nodeName === '#text') { if (v.nodeName === '#text') {
const text = v.data;
if (text.includes('\\[') || text.includes('\\(') || text.includes('$$')) {
return <Latex key={key} delimiters={latexDelimiters}>{text}</Latex>;
}
// Only wrap top-level text nodes in <p> // Only wrap top-level text nodes in <p>
if (keyPrefix === '' && v.data.trim() !== '') { if (keyPrefix === '' && v.data.trim() !== '') {
return <p key={key}>{v.data}</p>; return <p key={key}>{v.data}</p>;
@@ -120,10 +106,6 @@ function Article({ cache }) {
return null; return null;
} }
if (DANGEROUS_TAGS.includes(v.localName)) {
return <span key={key} dangerouslySetInnerHTML={{ __html: v.outerHTML }} />;
}
const Tag = v.localName; const Tag = v.localName;
if (isCodeBlock(v)) { if (isCodeBlock(v)) {
return ( return (
@@ -134,11 +116,6 @@ function Article({ cache }) {
); );
} }
const textContent = v.textContent.trim();
const isMath = (textContent.startsWith('\\(') && textContent.endsWith('\\)')) ||
(textContent.startsWith('\\[') && textContent.endsWith('\\]')) ||
(textContent.startsWith('$$') && textContent.endsWith('$$'));
const props = { key: key }; const props = { key: key };
if (v.hasAttributes()) { if (v.hasAttributes()) {
for (const attr of v.attributes) { for (const attr of v.attributes) {
@@ -147,21 +124,6 @@ function Article({ cache }) {
} }
} }
if (isMath) {
let mathContent = v.textContent;
// align environment requires display math mode
if (mathContent.includes('\\begin{align')) {
const trimmed = mathContent.trim();
if (trimmed.startsWith('\\(')) {
// Replace \( and \) with \[ and \] to switch to display mode
const firstParen = mathContent.indexOf('\\(');
const lastParen = mathContent.lastIndexOf('\\)');
mathContent = mathContent.substring(0, firstParen) + '\\[' + mathContent.substring(firstParen + 2, lastParen) + '\\]' + mathContent.substring(lastParen + 2);
}
}
return <Tag {...props}><Latex delimiters={latexDelimiters}>{mathContent}</Latex></Tag>;
}
if (VOID_ELEMENTS.includes(Tag)) { if (VOID_ELEMENTS.includes(Tag)) {
return <Tag {...props} />; return <Tag {...props} />;
} }
+11 -33
View File
@@ -8,19 +8,9 @@ function Feed({ updateCache }) {
const [stories, setStories] = useState(() => JSON.parse(localStorage.getItem('stories')) || false); const [stories, setStories] = useState(() => JSON.parse(localStorage.getItem('stories')) || false);
const [error, setError] = useState(''); const [error, setError] = useState('');
const [loadingStatus, setLoadingStatus] = useState(null); const [loadingStatus, setLoadingStatus] = useState(null);
const [filterSmallweb, setFilterSmallweb] = useState(() => localStorage.getItem('filterSmallweb') === 'true');
const handleFilterChange = e => {
const isChecked = e.target.checked;
setStories(false);
setFilterSmallweb(isChecked);
localStorage.setItem('filterSmallweb', isChecked);
};
useEffect(() => { useEffect(() => {
const controller = new AbortController(); fetch('/api')
fetch(filterSmallweb ? '/api?smallweb=true' : '/api', { signal: controller.signal })
.then(res => { .then(res => {
if (!res.ok) { if (!res.ok) {
throw new Error(`Server responded with ${res.status} ${res.statusText}`); throw new Error(`Server responded with ${res.status} ${res.statusText}`);
@@ -36,19 +26,21 @@ function Feed({ updateCache }) {
if (!updated) return; if (!updated) return;
if (!stories || !stories.length) {
setStories(newApiStories);
localStorage.setItem('stories', JSON.stringify(newApiStories));
}
setLoadingStatus({ current: 0, total: newApiStories.length }); setLoadingStatus({ current: 0, total: newApiStories.length });
let currentStories = Array.isArray(stories) ? [...stories] : []; let currentStories = Array.isArray(stories) ? [...stories] : [];
let preloadedCount = 0; let preloadedCount = 0;
for (const [index, newStory] of newApiStories.entries()) { for (const [index, newStory] of newApiStories.entries()) {
if (controller.signal.aborted) {
break;
}
try { try {
const storyFetchController = new AbortController(); const controller = new AbortController();
const timeoutId = setTimeout(() => storyFetchController.abort(), 10000); // 10-second timeout const timeoutId = setTimeout(() => controller.abort(), 10000); // 10-second timeout
const storyRes = await fetch('/api/' + newStory.id, { signal: storyFetchController.signal }); const storyRes = await fetch('/api/' + newStory.id, { signal: controller.signal });
clearTimeout(timeoutId); clearTimeout(timeoutId);
if (!storyRes.ok) { if (!storyRes.ok) {
@@ -97,17 +89,11 @@ function Feed({ updateCache }) {
setLoadingStatus(null); setLoadingStatus(null);
}, },
(error) => { (error) => {
if (error.name === 'AbortError') {
console.log('Feed fetch aborted.');
return;
}
const errorMessage = `Failed to fetch the main story list from the API. Your connection may be down or the server might be experiencing issues. ${error.toString()}.`; const errorMessage = `Failed to fetch the main story list from the API. Your connection may be down or the server might be experiencing issues. ${error.toString()}.`;
setError(errorMessage); setError(errorMessage);
} }
); );
}, [updateCache]);
return () => controller.abort();
}, [updateCache, filterSmallweb]);
return ( return (
<div className='container'> <div className='container'>
@@ -115,12 +101,7 @@ function Feed({ updateCache }) {
<title>QotNews</title> <title>QotNews</title>
<meta name="robots" content="index" /> <meta name="robots" content="index" />
</Helmet> </Helmet>
{loadingStatus && <p>Preloading stories {loadingStatus.current} / {loadingStatus.total}...</p>}
<div style={{marginBottom: '1rem'}}>
<input type="checkbox" id="filter-smallweb" className="checkbox" checked={filterSmallweb} onChange={handleFilterChange} />
<label htmlFor="filter-smallweb">Only Smallweb</label>
</div>
{error && {error &&
<details style={{marginBottom: '1rem'}}> <details style={{marginBottom: '1rem'}}>
<summary>Connection error? Click to expand.</summary> <summary>Connection error? Click to expand.</summary>
@@ -128,7 +109,6 @@ function Feed({ updateCache }) {
{stories && <p>Loaded feed from cache.</p>} {stories && <p>Loaded feed from cache.</p>}
</details> </details>
} }
{stories ? {stories ?
<div> <div>
{stories.map(x => {stories.map(x =>
@@ -150,8 +130,6 @@ function Feed({ updateCache }) {
: :
<p>Loading...</p> <p>Loading...</p>
} }
{loadingStatus && <p>Preloading stories {loadingStatus.current} / {loadingStatus.total}...</p>}
</div> </div>
); );
} }
+1 -27
View File
@@ -1,7 +1,6 @@
import React, { useState, useEffect } from 'react'; import React, { useState, useEffect } from 'react';
import { Link, useLocation, useHistory } from 'react-router-dom'; import { Link, useLocation } from 'react-router-dom';
import { Helmet } from 'react-helmet'; import { Helmet } from 'react-helmet';
import queryString from 'query-string';
import { sourceLink, infoLine, logos } from './utils.js'; import { sourceLink, infoLine, logos } from './utils.js';
import AbortController from 'abort-controller'; import AbortController from 'abort-controller';
@@ -9,19 +8,6 @@ function Results() {
const [stories, setStories] = useState(false); const [stories, setStories] = useState(false);
const [error, setError] = useState(false); const [error, setError] = useState(false);
const location = useLocation(); const location = useLocation();
const history = useHistory();
const handleFilterChange = e => {
const isChecked = e.target.checked;
const currentQuery = queryString.parse(location.search);
if (isChecked) {
currentQuery.article = 'true';
} else {
delete currentQuery.article;
}
history.push('/search?' + queryString.stringify(currentQuery));
};
useEffect(() => { useEffect(() => {
const controller = new AbortController(); const controller = new AbortController();
@@ -46,19 +32,11 @@ function Results() {
}; };
}, [location.search]); }, [location.search]);
const searchInArticle = queryString.parse(location.search).article === 'true';
return ( return (
<div className='container'> <div className='container'>
<Helmet> <Helmet>
<title>Search Results | QotNews</title> <title>Search Results | QotNews</title>
</Helmet> </Helmet>
<div style={{marginBottom: '1rem'}}>
<input type="checkbox" id="search-in-article" className="checkbox" checked={searchInArticle} onChange={handleFilterChange} />
<label htmlFor="search-in-article">Search in article</label>
</div>
{error && <p>Connection error?</p>} {error && <p>Connection error?</p>}
{stories ? {stories ?
<> <>
@@ -78,10 +56,6 @@ function Results() {
</div> </div>
{infoLine(x)} {infoLine(x)}
{!!x?._formatted &&
<p>{x._formatted.text.replace(/\n/g, ' ')}</p>
}
</div> </div>
) )
: :
+2 -6
View File
@@ -15,9 +15,7 @@ function Search() {
const newSearch = event.target.value; const newSearch = event.target.value;
setSearch(newSearch); setSearch(newSearch);
if (newSearch.length >= 3) { if (newSearch.length >= 3) {
const currentQuery = queryString.parse(location.search); const searchQuery = queryString.stringify({ 'q': newSearch });
currentQuery.q = newSearch;
const searchQuery = queryString.stringify(currentQuery);
history.replace('/search?' + searchQuery); history.replace('/search?' + searchQuery);
} else { } else {
history.replace('/'); history.replace('/');
@@ -26,9 +24,7 @@ function Search() {
const searchAgain = (event) => { const searchAgain = (event) => {
event.preventDefault(); event.preventDefault();
const currentQuery = queryString.parse(location.search); const searchString = queryString.stringify({ 'q': event.target[0].value });
currentQuery.q = event.target[0].value;
const searchString = queryString.stringify(currentQuery);
history.push('/search?' + searchString); history.push('/search?' + searchString);
inputRef.current.blur(); inputRef.current.blur();
} }
+1 -10
View File
@@ -11,8 +11,7 @@
border: 1px solid #828282; border: 1px solid #828282;
} }
.black .menu button, .black button {
.black .story-text button {
background-color: #444444; background-color: #444444;
border-color: #bbb; border-color: #bbb;
color: #ddd; color: #ddd;
@@ -67,11 +66,3 @@
.black .comment.lined { .black .comment.lined {
border-left: 1px solid #444444; border-left: 1px solid #444444;
} }
.black .checkbox:checked + label::after {
border-color: #ddd;
}
.black .copy-button {
color: #828282;
}
+1 -10
View File
@@ -11,8 +11,7 @@
border: 1px solid #828282; border: 1px solid #828282;
} }
.dark .menu button, .dark button {
.dark .story-text button {
background-color: #444444; background-color: #444444;
border-color: #bbb; border-color: #bbb;
color: #ddd; color: #ddd;
@@ -63,11 +62,3 @@
.dark .comment.lined { .dark .comment.lined {
border-left: 1px solid #444444; border-left: 1px solid #444444;
} }
.dark .checkbox:checked + label::after {
border-color: #ddd;
}
.dark .copy-button {
color: #828282;
}
-49
View File
@@ -189,13 +189,6 @@ span.source {
.comments { .comments {
margin-left: -1.25rem; margin-left: -1.25rem;
margin-top: 0;
margin-bottom: 0;
padding: 0;
}
.comments dl, .comments dd {
margin: 0;
} }
.comment { .comment {
@@ -312,50 +305,8 @@ button.comment {
.copy-button { .copy-button {
font: 1.5rem/1 'icomoon2'; font: 1.5rem/1 'icomoon2';
color: #828282;
background: transparent; background: transparent;
border: none; border: none;
cursor: pointer; cursor: pointer;
vertical-align: middle; vertical-align: middle;
} }
.checkbox {
-webkit-appearance: none;
appearance: none;
position: absolute;
opacity: 0;
cursor: pointer;
height: 0;
width: 0;
}
.checkbox + label {
position: relative;
cursor: pointer;
padding-left: 1.75rem;
user-select: none;
}
.checkbox + label::before {
content: '';
position: absolute;
left: 0;
top: 0.1em;
width: 1rem;
height: 1rem;
border: 1px solid #828282;
background-color: transparent;
border-radius: 3px;
}
.checkbox:checked + label::after {
content: "";
position: absolute;
left: 0.35rem;
top: 0.2em;
width: 0.3rem;
height: 0.6rem;
border: solid #000;
border-width: 0 2px 2px 0;
transform: rotate(45deg);
}
+1 -14
View File
@@ -20,8 +20,7 @@
background-color: #690000; background-color: #690000;
} }
.red .menu button, .red button {
.red .story-text button {
background-color: #440000; background-color: #440000;
border-color: #b00; border-color: #b00;
color: #b00; color: #b00;
@@ -81,15 +80,3 @@
.red .dot { .red .dot {
background-color: #440000; background-color: #440000;
} }
.red .checkbox + label::before {
border: 1px solid #690000;
}
.red .checkbox:checked + label::after {
border-color: #aa0000;
}
.red .copy-button {
color: #690000;
}
-19
View File
@@ -3169,11 +3169,6 @@ commander@^2.11.0, commander@^2.20.0:
resolved "https://registry.yarnpkg.com/commander/-/commander-2.20.3.tgz#fd485e84c03eb4881c20722ba48035e8531aeb33" resolved "https://registry.yarnpkg.com/commander/-/commander-2.20.3.tgz#fd485e84c03eb4881c20722ba48035e8531aeb33"
integrity sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ== integrity sha512-GpVkmM8vF2vQUkj2LvZmD35JxeJOLCwJ9cUkugyk2nuhbv3+mJvpLYYt+0+USMxE+oj+ey/lJEnhZw75x/OMcQ==
commander@^8.3.0:
version "8.3.0"
resolved "https://registry.yarnpkg.com/commander/-/commander-8.3.0.tgz#4837ea1b2da67b9c616a67afbb0fafee567bca66"
integrity sha512-OkTL9umf+He2DZkUq8f8J9of7yL6RJKI24dVITBmNfZBmri9zYZQrKkuXiKhyfPSu8tUhnVBB1iKXevvnlR4Ww==
commander@~2.19.0: commander@~2.19.0:
version "2.19.0" version "2.19.0"
resolved "https://registry.yarnpkg.com/commander/-/commander-2.19.0.tgz#f6198aa84e5b83c46054b94ddedbfed5ee9ff12a" resolved "https://registry.yarnpkg.com/commander/-/commander-2.19.0.tgz#f6198aa84e5b83c46054b94ddedbfed5ee9ff12a"
@@ -6861,13 +6856,6 @@ jsx-ast-utils@^2.1.0, jsx-ast-utils@^2.2.1:
array-includes "^3.1.1" array-includes "^3.1.1"
object.assign "^4.1.0" object.assign "^4.1.0"
katex@^0.16.0, katex@^0.16.25:
version "0.16.25"
resolved "https://registry.yarnpkg.com/katex/-/katex-0.16.25.tgz#61699984277e3bdb3e89e0e446b83cd0a57d87db"
integrity sha512-woHRUZ/iF23GBP1dkDQMh1QBad9dmr8/PAwNA54VrSOVYgI12MAcE14TqnDdQOdzyEonGzMepYnqBMYdsoAr8Q==
dependencies:
commander "^8.3.0"
killable@^1.0.0: killable@^1.0.0:
version "1.0.1" version "1.0.1"
resolved "https://registry.yarnpkg.com/killable/-/killable-1.0.1.tgz#4c8ce441187a061c7474fb87ca08e2a638194892" resolved "https://registry.yarnpkg.com/killable/-/killable-1.0.1.tgz#4c8ce441187a061c7474fb87ca08e2a638194892"
@@ -9131,13 +9119,6 @@ react-is@^16.13.1, react-is@^16.6.0, react-is@^16.7.0, react-is@^16.8.4:
resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.1.tgz#789729a4dc36de2999dc156dd6c1d9c18cea56a4" resolved "https://registry.yarnpkg.com/react-is/-/react-is-16.13.1.tgz#789729a4dc36de2999dc156dd6c1d9c18cea56a4"
integrity sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ== integrity sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==
react-latex-next@^3.0.0:
version "3.0.0"
resolved "https://registry.yarnpkg.com/react-latex-next/-/react-latex-next-3.0.0.tgz#3e347a13b1e701439b5fa52f75201bc86166854e"
integrity sha512-x70f1b1G7TronVigsRgKHKYYVUNfZk/3bciFyYX1lYLQH2y3/TXku3+5Vap8MDbJhtopePSYBsYWS6jhzIdz+g==
dependencies:
katex "^0.16.0"
react-router-dom@^5.0.1: react-router-dom@^5.0.1:
version "5.3.4" version "5.3.4"
resolved "https://registry.yarnpkg.com/react-router-dom/-/react-router-dom-5.3.4.tgz#2ed62ffd88cae6db134445f4a0c0ae8b91d2e5e6" resolved "https://registry.yarnpkg.com/react-router-dom/-/react-router-dom-5.3.4.tgz#2ed62ffd88cae6db134445f4a0c0ae8b91d2e5e6"