Punycode

Background

Messing with RSS feed recently (as NextJS do not have any official RSS parser plugin) , I realised Chinese URL links generated were not validated. UTF-8 encoded Chinese characters were not considered as valid.

What I've Learned

Punycode is a special encoding used to convert Unicode characters to ASCII, which is a smaller, restricted character set. Punycode is used to encode internationalized domain names (IDN).

https://www.punycoder.com/

How to recreate the problem?

Punycode is available to encode strings of Unicode to ASCII symbols, however, the returned URL is not working as expected. For example, a Chinese blog link like https://hk.news.yahoo.com/65個指明地方納入強制檢測公告-004959684.html was converted into https://hk.news.yahoo.com/xn--65-q43c84c9a92hp8grphq2u14i5uhd3aq94bo6lkm8c-004959684.html instead of https://hk.news.yahoo.com/65%E5%80%8B%E6%8C%87%E6%98%8E%E5%9C%B0%E6%96%B9%E7%B4%8D%E5%85%A5%E5%BC%B7%E5%88%B6%E6%AA%A2%E6%B8%AC%E5%85%AC%E5%91%8A-004959684.html

Action Items

Following code does not work as expected. Only the variable was converted instead of the whole URI string.

fs.readdirSync(blogPostDir)

  .map((fileName) => {
    // we need the full path of the file to be able to read it

    const fullPath = path.join(blogPostDir, fileName);

    // read the file so we can grab the front matter

    const file = fs.readFileSync(fullPath, "utf8");

    // for the RSS feed we don't need the html, we

    // just want the attributes

    const { data: frontmatter, content } = matter(file);

    const excerpt = getExcerpt(content, 800);

    const url = CONFIG.URL + `/posts/` + fileName.replace(".md", "") + `/`;

    // console.log(excerpt);

    // I want access to the fileName later on so we save it to our object

    return { ...frontmatter, fileName, excerpt, url };
  })

  // sort the items by date in descending order, feel free

  // to customize this as needed to sort your RSS items properly

  .filter((post) => post.draft === false)

  .sort((a, b) => +new Date(b.date) - +new Date(a.date))

  // loop over each blog post and add it to our RSS feed

  .forEach(({ title, date, tags, category, fileName, excerpt, url }) => {
    // title, description, and date are properties of my front matter

    // attributes. Yours might be different depending on your data structure

    feed.item({
      title,

      description: excerpt,

      url: punycode.encode(url),

      author: CONFIG.AUTHOR_NAME,

      categories: tags,

      date,
    });
  });