The National Football League (NFL) continues to shed viewers. Ratings have been down for each of the first three weeks of the 2016 season. For 2015, the NFL averaged 20.1 million viewers. This year, it has averaged only 18.3 million, with the marquee Sunday night and Monday night games each down 12 percent.

And the trend continues to be down. For example, 20.6 million people tuned in for last week’s Sunday Night Football telecast between the Dallas Cowboys and Chicago Bears. This is down from 22.8 million viewers the previous week and 23.1 million for the opening week. The early-afternoon games on CBS dropped by 18 percent compared to the third week of last season.

So why is the NFL–the most watched programming on TV–losing viewers?

The NFL and its defenders say that it has to do with lackluster games and the absence of two of the league’s biggest stars–Peyton Manning (retired) and Tom Brady (serving a four game suspension). Still others, such as Forbes columnist Mike Ozanian, think it is because of the Black Lives Matter protest started by Colin Kaepernick. Originally, the backup mulatto quarterback of the San Francisco 49ers sat during the national anthem that precedes each game. Now Mr. Kaepernick takes a knee during the anthem, with several other players from various teams joining him. Still other players give the black-power salute. All the protesters so far have been black. Mr. Ozanian writes:

The downturn in ratings has occurred recently even though the league’s broadcasters like Fox and CBS do not seem to be mentioning Colin Kaepernick and the anthem boycott as much as they did at the beginning. To me, this implies that the low ratings are here to stay unless the players stop protesting the anthem.

The protest has no doubt spurred some people to turn off the NFL, as urged by popular Twitter hashtags such as #BoycottNFL and #BoycottTheNFL. Even celebrities such as actor James Woods and radio host Joe Walsh have joined the boycott.

But could it also be that whites are getting sick of watching an increasingly black league that seems to exclude talented white players? In 1985, the league was nearly half white, with blacks making up 52 percent of the rosters. Today, blacks are 67 percent of players with whites at only 31 percent. There are very few Asians or Hispanics in the NFL, though Pacific Islanders–especially Samoans–seem to be well represented for their small population.

This does not match the NFL fan base. Seventy-seven percent of the fans are white, 15 percent are black, and 8 percent are Hispanic. Moreover, it is an older audience, with 37 percent of NFL fans over 55.

Perhaps these older white fans started following football when the demographics were not so skewed. Younger generations of whites don’t seem as enthralled by the mostly-black league. Only 20 percent of NFL fans are in the 18-34 age bracket. That number slips to 9 percent for viewers 17 and younger.

The website Caste Football is dedicated to race and sports and makes a compelling case that many white athletes are excluded from college and pro football simply because of their race. But even if this were not the case, why should whites–or other non-blacks–follow a mostly black league? Blacks don’t follow hockey; only three percent of NHL fans are black.

The National Basketball Association (NBA) has already felt a ratings slip due to demographics. The league composition fluctuates around about 75 percent black. Consequently, there are more black NBA fans (45 percent) than white (40 percent).

Demographics alone are probably driving some whites away from football. Combine that with outright racial activism such as black power salutes during the national anthem and it is no surprise that more and more whites may be giving up on the NFL.

So what has the league’s official response been to the Black Lives Matter-inspired protests? Here are just a few gems:

I support our players speaking out on issues they think need to be changed in our society. We don’t live in a perfect society. Our players have strong views about things so I support our players speaking out about that.

—Roger Goodell, NFL Commissioner

We recognize and respect Kap’s decision and his constitutional rights to do what he’s doing and it sounds like it’s been a positive change. There’s been a lot of positive things that have come out of it.

—Chip Kelly, San Francisco 49ers head coach

It’s really raised consciousness and it’s a wonderful thing.

—Steve Young, ESPN broadcaster

Of course, ‘raising consciousness” and “speaking out” are one-way streets. White baseball player Steve Clevenger of the Seattle Mariners sent the following tweets from his personal account in the wake of the Charlotte riots:

Mr. Clevenger was promptly suspended without pay by the Mariners and his baseball career is probably over. White pundit Skip Bayless of Fox Sports set the tone for the media treatment of Mr. Clevenger:

I am appalled that another white American–a professional athlete no less–is capable of typing these thoughts in a public forum, in two different tweets, and actually hitting “send.” You want to talk about showing your true colors in public?

Meanwhile, Mr. Kaepernick was featured on the cover of TIME magazine and still has a well-paid job. His jersey is the top seller in the NFL.

I have long thought our side should boycott.

The #BoycottNFL movement gives us a great place to start.