Crawled - currently not indexed: the cause was 173 words

Search Console gave me this on a page I was fairly sure was fine:

Page indexing
  Page is not indexed: Crawled - currently not indexed
Discovery
  Sitemaps:       No referring sitemaps detected
  Referring page: None detected
Crawl
  Crawl allowed?  Yes
  Page fetch:     Successful
Indexing
  User-declared canonical: N/A
  Google-selected canonical: N/A

Crawl allowed, fetch successful, and then a refusal. That combination is worth understanding, because it rules out most of what people check first.

What it is not#

Crawl allowed: Yes and Page fetch: Successful mean robots.txt is not blocking anything and the server returned the page. I confirmed both anyway:

curl -s https://kartik.contentcalendar.in/robots.txt
# User-Agent: *
# Allow: /
# Disallow: /admin
# Sitemap: https://kartik.contentcalendar.in/sitemap.xml

The sitemap was live and valid. Canonicals were present and self-referencing on every page. None of it mattered, because "Crawled - currently not indexed" is not an error. It is a judgement. Google fetched the page, looked at it, and decided it was not worth an index slot.

Measuring instead of guessing#

The useful question is what Googlebot actually received. Not what the browser shows — the browser shows you the page after JavaScript has run, which is precisely the view that hides this problem.

So: fetch the raw HTML, strip scripts and styles, count what is left.

import re, urllib.request
 
for path in ['/', '/web', '/ai', '/editing']:
    html = urllib.request.urlopen('https://example.com' + path).read().decode('utf-8', 'ignore')
    body = re.sub(r'(?is)<script.*?</script>', ' ', html)
    body = re.sub(r'(?is)<style.*?</style>', ' ', body)
    body = re.sub(r'(?s)<[^>]+>', ' ', body)
    body = re.sub(r'\s+', ' ', body).strip()
    print(f'{path:<10} {len(body.split()):>5} words')

The result:

There it is. The homepage was fine. Every portfolio page was serving under 300 words — and those pages are supposed to be the ones showing the work.

Why#

The grids looked like this:

const WebPortfolio = () => {
  const [projects, setProjects] = useState([]);
  const [loading, setLoading] = useState(true);
 
  useEffect(() => {
    const fetchProjects = async () => {
      const data = await getProjects();          // Firestore, in the browser
      setProjects(data.filter((p) => p.section === 'web'));
      setLoading(false);
    };
    fetchProjects();
  }, []);
  // ...
};

Standard client-side fetching, and the component works perfectly. But the initial state is [], so the server renders zero cards. Every project title, category and client name — the entire substance of the page — appeared only after JavaScript executed and a network round-trip to Firestore completed.

Google does render JavaScript. But rendering is a second pass on Google's own schedule, and a page has to look worth the resources on the first pass to earn it. A near-empty page can be judged before its content ever exists.

The fix#

Nothing exotic: give the component its data on the server, and let the client fetch refresh it.

// src/app/web/page.jsx  - a server component
import seedProjects from '../../data/portfolio.json';
 
export default function WebPortfolioPage() {
  return <WebPortfolio initialProjects={seedProjects.filter((p) => p.section === 'web')} />;
}

// the grid - two lines changed
const WebPortfolio = ({ initialProjects = [] }) => {
  const [projects, setProjects] = useState(initialProjects);
  const [loading, setLoading] = useState(initialProjects.length === 0);
  // the existing useEffect still runs and replaces this with live data

The useEffect is untouched. Firestore still owns the live data. The only change is that the first paint — and therefore the HTML response — is no longer empty.

Verified by counting project titles in the raw HTML rather than trusting the word count alone:

web = [p for p in portfolio if p['section'] == 'web']
html = urllib.request.urlopen('http://localhost:3000/web').read().decode()
for record in web:
    print('IN HTML' if record['title'] in html else 'MISSING', record['title'])

Six of six.

The other half: the sitemap was never submitted#

No referring sitemaps detected had me checking the sitemap repeatedly. It was fine every time.

That field does not report whether a valid sitemap exists. It reports whether Google found this URL through a sitemap submitted in Search Console. Mine had never been submitted. A sitemap at a conventional path is discoverable in principle, but submitting it explicitly is what creates the association that field is describing.

Search Console → Sitemaps → sitemap.xml → Submit. That is the entire fix, and no amount of work in the repository substitutes for it.

What this demonstrates#

The diagnostic mistake I nearly made was treating "not indexed" as a configuration bug. Configuration bugs are satisfying to hunt: there is a file, it has a wrong line, you fix the line. So I checked robots.txt, then canonicals, then the sitemap, then trailing-slash redirects — all correct, all irrelevant.

"Crawled - currently not indexed" is Google declining to spend an index slot. The question it is answering is whether the page is worth indexing, and the only way to argue with that is to make the page carry something.

Two habits came out of this that I would keep:

Measure the response, not the page. curl and count words. The browser renders your JavaScript for you, which is exactly the help you do not want when diagnosing what a crawler received.

Read Search Console's fields literally. "No referring sitemaps" means no submitted sitemap referred this URL. It does not mean the sitemap is broken. Every minute I spent re-validating the XML was a minute spent on a field I had misread.