Background
Messing with RSS feed recently (as NextJS do not have any official RSS parser plugin) , I realised Chinese URL links generated were not validated. UTF-8 encoded Chinese characters were not considered as valid.
What I've Learned
Punycode is a special encoding used to convert Unicode characters to ASCII, which is a smaller, restricted character set. Punycode is used to encode internationalized domain names (IDN).
https://www.punycoder.com/
How to recreate the problem?
Punycode is available to encode strings of Unicode to ASCII symbols, however, the returned URL is not working as expected. For example, a Chinese blog link like https://hk.news.yahoo.com/65個指明地方納入強制檢測公告-004959684.html was converted into https://hk.news.yahoo.com/xn--65-q43c84c9a92hp8grphq2u14i5uhd3aq94bo6lkm8c-004959684.html instead of https://hk.news.yahoo.com/65%E5%80%8B%E6%8C%87%E6%98%8E%E5%9C%B0%E6%96%B9%E7%B4%8D%E5%85%A5%E5%BC%B7%E5%88%B6%E6%AA%A2%E6%B8%AC%E5%85%AC%E5%91%8A-004959684.html
Action Items
Following code does not work as expected. Only the variable was converted instead of the whole URI string.
fs.readdirSync(blogPostDir)
.map((fileName) => {
// we need the full path of the file to be able to read it
const fullPath = path.join(blogPostDir, fileName);
// read the file so we can grab the front matter
const file = fs.readFileSync(fullPath, "utf8");
// for the RSS feed we don't need the html, we
// just want the attributes
const { data: frontmatter, content } = matter(file);
const excerpt = getExcerpt(content, 800);
const url = CONFIG.URL + `/posts/` + fileName.replace(".md", "") + `/`;
// console.log(excerpt);
// I want access to the fileName later on so we save it to our object
return { ...frontmatter, fileName, excerpt, url };
})
// sort the items by date in descending order, feel free
// to customize this as needed to sort your RSS items properly
.filter((post) => post.draft === false)
.sort((a, b) => +new Date(b.date) - +new Date(a.date))
// loop over each blog post and add it to our RSS feed
.forEach(({ title, date, tags, category, fileName, excerpt, url }) => {
// title, description, and date are properties of my front matter
// attributes. Yours might be different depending on your data structure
feed.item({
title,
description: excerpt,
url: punycode.encode(url),
author: CONFIG.AUTHOR_NAME,
categories: tags,
date,
});
});